-
Notifications
You must be signed in to change notification settings - Fork 96
/
rbe_repo.bzl
1254 lines (1124 loc) · 60.1 KB
/
rbe_repo.bzl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Copyright 2017 The Bazel Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Repository Rules to pick/generate toolchain configs for a container image.
The toolchain configs (+ platform) produced/selected by this rule can be used
to, e.g., run a remote build in which remote actions will run inside a
container image.
Exposes the rbe_autoconfig macro that encapsulates all functionality to
create and use toolchain configs. The main use cases for this rule are
1. If you use the rbe-ubuntu 16_04 image in your RBE builds: This macro
enables automatic selection of toolchain configs for your RBE build.
As long as you are using a release version of Bazel and have your pin
to the bazel-toolchains repo up to date, this rule should "just work"
to select toolchain configs that have been generated for you before hand.
More details about the rbe-ubuntu 16_04 image: https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04
2. If you use a container that extends from the rbe-ubuntu 16_04 image
in your RBE builds: This macro allows you to define which version of
the rbe-ubuntu 16_04 image you built yours from, and then it will either
pick toolchain configs in the bazel-toolchains repo that work for you,
or will generate them on the fly by pulling the rbe-ubuntu 16_04
container you used as base and running some commands inside the container.
The main reason for having to generate a toolchain config (as opposed to
using one that is checked-in the bazel-toolchains repo) is due to your
base rbe-ubuntu 16_04 image not being compatible with the latest one,
for the given version of Bazel you are using.
If you want to make sure you can use checked-in configs, you should
rebuild your container, using the latest rbe-ubuntu 16_04 image as base,
whenever you udpate Bazel versions.
3. If you use a custom container that does not extend from the rbe-ubuntu 16_04
image (or your project has custom configure like repo rules, or you
need to run your configure like repo rules with different environment
settings):
3.1. If you don't mind having to generate configs each time you run Bazel from
a clean client, then this rule can do just that, by simply specifying the
relevant information about which container to use.
3.2. If you don't want to generate configs every time, you can use
rbe_autoconfig to setup a "toolchain_config_repo" from which anyone
that builds on RBE with your container can pull pre-generated configs from.
To do this, rbe_autoconfig allows specification of a 'toolchain_config_suite_spec'.
A 'toolchain_config_suite_spec' specifies all details of an external repo
that will be used to both export to and read from toolchain configs.
Details for setting up a "toolchain_config_repo" are below.
More about configure like repo rules: https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-configure-style-workspace-rules
If rbe_autoconfig needs to generate toolchain configs, the process is as follows:
- Pull the selected toolchain container image (using 'docker pull').
- Start up a container using the pulled image, copying either a small sample
project or the current project (if output_base is set).
- Install the current version of Bazel (one currently running) on the container
(or the one passed in with optional attr). Container must have tools required to install
and run Bazel (i.e., a jdk, a C/C++ compiler, a python interpreter).
- Run a bazel command to build the local_config_cc remote repository inside the container.
- Extract local_config_cc produced files (inside the container) to the produced
remote repository.
- Produce a default BUILD file with platform and toolchain targets to use the container
in a remote build.
- Optionally copies the local_config_cc produced files to the project srcs under the
given output_base directory.
For use case 1. If you use the rbe-ubuntu 16_04 image in your RBE builds,
add to your WORKSPACE file the following:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
http_archive(
name = "bazel_toolchains",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/archive/<latest_release>.tar.gz",
"https://github.com/bazelbuild/bazel-toolchains/archive/<latest_release>.tar.gz",
],
strip_prefix = "bazel-toolchains-<latest_commit>",
sha256 = "<sha256>",
)
load(
"@bazel_toolchains//repositories:repositories.bzl",
bazel_toolchains_repositories = "repositories",
)
bazel_toolchains_repositories()
load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig")
# This is the simplest form of calling rbe_autoconfig.
# See below other examples, but your WORKSPACE should likely
# only need one single rbe_autoconfig target
rbe_autoconfig(
name = "rbe_default",
# If you want your rbe_autoconfig rule to never pull a container
# and fail fast if toolchain configs are not available, uncomment the
# following line:
# use_checked_in_confs = "Force",
)
For the recommended value of <latest_release> please see
https://releases.bazel.build/bazel-toolchains.html
For use case 2. If you use a container that extends from the rbe-ubuntu 16_04
image, add to your WORKSPACE file the following:
<Add the bazel-toolchains repo http_archive, same as above>
rbe_autoconfig(
name = "rbe_default",
base_container_digest = <SHA of rbe-ubuntu 16_04 you used as base>
registry = "gcr.io",
repository = "your-project/your-image-name",
# Digest is recommended for any use case other than testing.
digest = "sha256:deadbeef",
)
For use case 3.1. If you are using a custom container, and don't mind
having to generate configs each time, add to your WORKSPACE file
the following:
<Add the bazel-toolchains repo http_archive, same as above>
rbe_autoconfig(
name = "rbe_your_custom_container",
registry = "gcr.io",
repository = "your-project/your-image-name",
# Digest is recommended for any use case other than testing.
digest = "sha256:deadbeef",
)
For use case 3.2. If you are using a custom container, and don't
want to generate configs every time. The setup is as follows:
Define a repo (can be the same where you host the sources you want
to build on RBE or a separate one) which will host your published
toolchain configs. We call this the "toolchain-config-repo" in the
below
Define a 'toolchain_config_suite_spec', which specifies a container
(repo + registry) and an output_base, the relative path, within
the toolchain-config-repo where toolchain configs will be published to.
You can only have one 'toolchain_config_suite_spec' per container for which
you will be producing toolchain configs for, but multiple versions of the same
container (i.e., with different sha / tags) can share the same
'toolchain_config_suite_spec'.
Also, the same 'toolchain_config_suite_spec' can also be used to host
multiple toolchain configs that vary in environment variables, and
additional config repos (i.e., repos corresponding to configure like
repository rules) that are needed for different types of builds. For example,
a configuration for msan (which needs specific env variables) and one for
default C++ builds can share an output_base.
Lastly, a 'toolchain_config_suite_spec' will store configs for several
versions of Bazel (any that you specify).
For detailed instructions of how to set up a 'toolchain_config_suite_spec'
please see //rules/rbe_repo/toolchain_config_suite_spec.bzl
Once you have set up the toolchain_config_suite_spec you can add to your
WORKSPACE the following:
<Add the bazel-toolchains repo http_archive, same as above>
load("//path/to_your/toolchain_config_suite_spec.bzl", "your_toolchain_config_suite_spec_struct")
rbe_autoconfig(
name = "rbe_your_custom_toolchain_config_suite_spec",
export_configs = True,
toolchain_config_suite_spec = your_toolchain_config_suite_spec_struct,
)
You can then run:
RBE_AUTOCONF_ROOT=$(pwd) bazel build @rbe_your_custom_toolchain_config_suite_spec//...
This will create the toolchain configs in the 'output_base' defined in the
'toolchain_config_suite_spec'. It will generate configs for the current version
of Bazel you are running with (overridable via attr).
This will also (abusing Bazel hermeticity principles) modify the versions.bzl
file in the 'output_base'. This is so that subsequent executions of the target
(by you, or by any of your users after you have checked-in these generated files)
will be able to directly use them without having to generate them again.
You should check-in your repo these changes so that the generated configs
are available to all other users of your repo.
For users of your 'toolchain_config_suite_spec' all that they need to do is
add to their WORKSPACE:
<Add the bazel-toolchains repo http_archive, same as above>
load("//path/to_your/toolchain_config_suite_spec.bzl", "your_toolchain_config_suite_spec_struct")
rbe_autoconfig(
name = "rbe_your_custom_toolchain_config_suite_spec",
toolchain_config_suite_spec = your_toolchain_config_suite_spec_struct,
)
And that's it! They should be able to get checked-in configs every time,
as long as, whenever there is a new Bazel needed for RBE builds:
- You, the owner of the 'toolchain_config_suite_spec' generates and
publishes the new toolchain configs to your repo. This is needed because
new Bazel versions can only be guaranteed to work with toolchain
configs that were generated for the specific version of Bazel used.
- The users of your 'toolchain_config_suite_spec' update their pin to
your repo.
If you want to create (more) different sets of toolchain configurations (a toolchain_config_spec)
with a different set of env variables, you can do so by reusing the
'toolchain_config_suite_spec', and providing a distinct 'toolchain_config_spec_name'. Example:
rbe_autoconfig(
name = "rbe_custom_env2",
env = {<dict declaring env variables>},
export_configs = True,
toolchain_config_spec_name = "<unique name to assign this toolchain_config_spec>",
toolchain_config_suite_spec = your_toolchain_config_suite_spec_struct,
)
rbe_autoconfig(
name = "rbe_custom_env2",
env = {<dict declaring env variables>},
export_configs = True,
toolchain_config_spec_name = "<unique name to assign this toolchain_config_spec>",
toolchain_config_suite_spec = your_toolchain_config_suite_spec_struct,
)
As of Bazel 0.29.0, platforms support exec_properties instead of the deprecated
remote_execution_properties to configure remote execution properties. The new
field is a string->string dictionary rather than a proto serialized as a
string.
rbe_autoconfig now has a field use_legacy_platform_definition, which for
backward compatibility reasons is set by default to True. Setting it to False
causes the underlying platform to be configured using the new exec_properties
field.
Furthermore, rbe_autoconfig itself also has an exec_properties field. Any
values set there are used in configuring the underlying platform. This field
only works if use_legacy_platform_definition is set to False.
Note that the container image cannot be set in rbe_autoconfig via the
exec_properties field.
Here is an example of an rbe_autoconfig that configures its underlying platform
to set the size of the shared memory partition for the docker container to 128
megabytes.
load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_rbe_exec_properties_dict")
rbe_autoconfig(
name = "rbe_default",
use_legacy_platform_definition = False,
exec_properties = create_rbe_exec_properties_dict(docker_shm_size = "128m"),
)
Note the use of create_rbe_exec_properties_dict. This is a Bazel macro that
makes it convenient to create the dicts used in exec_properties. You should
always prefer to use it over composing the dict manually.
Additionally, there are standard execution property dicts that you may want to
use. These standard dicts should always be preferred if defined. These standard
dicts are defined in a local repo that can be set up via a Bazel macro called
rbe_exec_properties. The following example has rbe_autoconfig create an
underlying platform that allows network access to the remote execution worker.
load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "rbe_exec_properties")
rbe_exec_properties(
name = "exec_properties",
)
load("@exec_properties//:constants.bzl", "NETWORK_ON")
rbe_autoconfig(
name = "rbe_default",
use_legacy_platform_definition = False,
exec_properties = NETWORK_ON,
)
For more information on create_rbe_exec_properties_dict, rbe_exec_properties
and other related Bazel macros, see https://github.com/bazelbuild/bazel-toolchains/tree/master/rules/exec_properties
NOTES:
READ CAREFULLY THROUGH THESE NOTES, NO MATTER YOUR USE CASE:
NOTE 1: SETTING TOOLCHAIN FLAGS
This is not an up to date source for flags, and just provides general guidance
please see //bazelrc/.latest.bazelrc for the most up to date flags.
Once you have added the rbe_autoconfig rule to your WORKSPACE, you will
need to set up toolchain flags that select the appropriate toolchain configs.
The flags below, show an sample of those flags, which was last reviewed
with Bazel 0.25.0 and for a rbe_autoconfig rule with name 'rbe_default'.
If you are using a later version of Bazel or your rbe_autoconfig target
has a different name, please adjust accordingly.
bazel build ... \
--crosstool_top=@rbe_default//cc:toolchain \
--host_javabase=@rbe_default//java:jdk \
--javabase=@rbe_default//java:jdk \
--host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8 \
--java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8 \
--extra_execution_platforms=@rbe_default//config:platform \
--host_platform=@rbe_default//config:platform \
--platforms=@rbe_default//config:platform \
--extra_toolchains=@rbe_default//config:cc-toolchain \
NOTE 2: WHEN DOES THIS RULE PULL A CONTAINER
Most users of rbe_autoconfig do not expect their target to pull a container.
If your rbe_autoconfig rule nevertheless pulls a container its because it
could not find checked-in configs that match:
1- The Bazel version you are currently using
2- The container you selected (if you are not setting base_container_digest)
3- The environment or config repos you requested
The simplest fix for 1 is to update your pin to the bazel-toolchains repo (and
to the source of your custom 'toolchain_config_suite_spec'). If that does not
fix the issue, you can try to use an older toolchain cofig by passing
an older version in the bazel_version 'attr' (which may or may not work with
the current version you are running). You should also contact the owners of
bazel-toolchains (or the custom 'toolchain_config_suite_spec' repo) to have them
publish configs for any new version if they have not done so.
The simplest fix for 2, is to rebuild your custom container using as base
the latest version of the base_container_digest. If that does not work, contact
the owners of the bazel-toolchains repo (i.e., create an issue in this repo)
or the owners of custom 'toolchain_config_suite_spec' repo if you are using one.
The only possible fix for 3, if you want this custom config spec to be supported
with checked-in configs, is to contact the owners of the bazel-toolchains repo
(or the custom 'toolchain_config_suite_spec' repo), to ask them to add this spec
to their WORKSPACE and generate configs for it.
NOTE 3: USE OF PROJECT ROOT
When this rule needs to export toolchain configs, or when it needs
to generate configs for custom config repos (i.e., corresponding to
configure like repo rules needed in your RBE build), This rule depends
on the value of the environment variable "RBEAUTOCONF_ROOT".
This env var should be set to point to the absolute path root of your project.
Use the full absolute path to the project root (i.e., no '~', '../', or
other special chars).
NOTE 4: PREREQUISITES FOR RUNNING THIS RULE
If this rule needs to generate configs, it expects the following
utilities to be installed and available on the PATH:
- docker
- tar
- bash utilities (e.g., cp, mv, rm, etc)
- docker authentication to pull the desired container should be set up
(rbe-ubuntu16-04 does not require any auth setup currently).
NOTE 5: HERMETICITY AND THIS RULE
Note this is a very not hermetic repository rule that can actually change the
contents of your project sources. While this is generally not recommended by
Bazel, its the only reasonable way to get a rule that can produce valid
toolchains / platforms that need to be made available to Bazel before execution
of any build actions, AND at the same time, make them available to other
users so they do not need to be regenerated again. This can be done, safely,
to a certain extent, because these containers are pulled and dealt with by
SHA, all the outputs produced should be completely independent of where they
were built and can be leveraged by all users of a container.
NOTE 6: KNOWN LIMITATIONS
- This rule can only run on Linux or Windows if it needs to generate configs.
- This rule uses Bazelisk to run Bazel inside the given container.
The container, thus, must be able to execute the Bazelisk binary
(i.e., Linux or Windows based container must be capable of running
the respective linux-amd or windows-amd releases from
https://github.com/bazelbuild/bazelisk/releases)
- If using export_configs, and you have multiple rbe_autoconfig targets
pointing to the same toolchain_config_suite_spec, these rules should not
be executed together in the same bazel command, as they all depend on
reading/writing to the same versions.bzl file.
- This rule cannot generate configs if: 1) it needs to pull additional
config repos and 2) the project's WORKSPACE contains local_repository
rules pointing to directories above the project root.
"""
load(
"//configs/dependency-tracking:ubuntu1604.bzl",
BAZEL_LATEST = "bazel",
)
load(
"//rules/rbe_repo:build_gen.bzl",
"create_alias_platform",
"create_config_aliases",
"create_export_platform",
"create_external_repo_platform",
"create_java_runtime",
)
load(
"//rules/rbe_repo:checked_in.bzl",
"CHECKED_IN_CONFS_FORCE",
"CHECKED_IN_CONFS_TRY",
"CHECKED_IN_CONFS_VALUES",
"validateUseOfCheckedInConfigs",
)
load(
"//rules/rbe_repo:container.bzl",
"get_java_home",
"get_java_version",
"pull_container_needed",
"pull_image",
"run_and_extract",
)
load(
"//rules/rbe_repo:outputs.bzl",
"create_configs_tar",
"create_versions_file",
"expand_outputs",
)
load(
"//rules/rbe_repo:toolchain_config_suite_spec.bzl",
"config_to_string_lists",
"default_toolchain_config_suite_spec",
"validate_toolchain_config_suite_spec",
)
load(
"//rules/rbe_repo:util.bzl",
"AUTOCONF_ROOT",
"DOCKER_PATH",
"copy_to_test_dir",
"os_family",
"print_exec_results",
"resolve_image_name",
"resolve_project_root",
"resolve_rbe_original_image_name",
"validate_host",
)
load(
"//rules/rbe_repo:version_check.bzl",
"extract_version_number",
"parse_rc",
)
# Version to fallback to if not provided explicitly and local is non-release version.
_BAZEL_VERSION_FALLBACK = BAZEL_LATEST
_CONFIG_REPOS = ["local_config_cc"]
_DEFAULT_TOOLCHAIN_CONFIG_SPEC_NAME = "default_toolchain_config_spec_name"
_EXEC_COMPAT_WITH = {
"Linux": [
"@bazel_tools//platforms:linux",
"@bazel_tools//platforms:x86_64",
"@bazel_tools//tools/cpp:clang",
],
"Windows": [
"@bazel_tools//platforms:windows",
"@bazel_tools//platforms:x86_64",
],
}
_TARGET_COMPAT_WITH = {
"Linux": [
"@bazel_tools//platforms:linux",
"@bazel_tools//platforms:x86_64",
],
"Windows": [
"@bazel_tools//platforms:windows",
"@bazel_tools//platforms:x86_64",
],
}
def _rbe_autoconfig_impl(ctx):
"""Core implementation of _rbe_autoconfig repository rule."""
bazel_version_debug = "Bazel %s" % ctx.attr.bazel_version
if ctx.attr.bazel_rc_version:
bazel_version_debug += " rc%s" % ctx.attr.bazel_rc_version
print("%s is used in %s." % (bazel_version_debug, ctx.attr.name))
if ctx.attr.use_checked_in_confs == CHECKED_IN_CONFS_FORCE and not ctx.attr.config_version:
fail(("Target '{name}' failed: use_checked_in_confs was set to '{force}' " +
"but no checked-in configs were found. " +
"Please check your pin to '@{toolchain_config_suite_spec_repo_name}' is up " +
"to date, and that you are using a release version of " +
"Bazel. You can also explicitly set the version of Bazel to " +
"an older version in the '{name}' rbe_autoconfig target " +
"which may or may not work with the version you are currently " +
"running with.").format(
name = ctx.attr.name,
force = CHECKED_IN_CONFS_FORCE,
toolchain_config_suite_spec_repo_name = ctx.attr.toolchain_config_suite_spec["repo_name"],
))
name = ctx.attr.name
image_name = resolve_image_name(ctx)
docker_tool_path = None
# Resolve default constraints if none set
target_compatible_with = ctx.attr.target_compatible_with
if not target_compatible_with:
target_compatible_with = _TARGET_COMPAT_WITH[os_family(ctx)]
exec_compatible_with = ctx.attr.exec_compatible_with
if not exec_compatible_with:
exec_compatible_with = _EXEC_COMPAT_WITH[os_family(ctx)]
# Resolve the paths to copy srcs to the container and to
# export configs.
mount_project_root, export_project_root, use_default_project = resolve_project_root(ctx)
# Check if pulling a container will be needed and pull it if so
digest = ctx.attr.digest
if pull_container_needed(ctx):
ctx.report_progress("validating host tools")
docker_tool_path = validate_host(ctx)
# Pull the image using 'docker pull'
pull_image(ctx, docker_tool_path, image_name)
# If tag is specified instead of digest, resolve it to digest in the
# image_name as it will be used later on in the platform targets.
if ctx.attr.tag:
result = ctx.execute([docker_tool_path, "inspect", "--format={{index .RepoDigests 0}}", image_name])
print_exec_results("Resolve image digest", result, fail_on_error = True)
image_name = result.stdout.splitlines()[0]
digest = image_name.split("@")[1]
print("Image with given tag `%s` is resolved to '%s', digest is '%s'" %
(ctx.attr.tag, image_name, digest))
# Get the value of JAVA_HOME to set in the produced
# java_runtime
java_home = None
java_version = None
if ctx.attr.create_java_configs:
java_home = get_java_home(ctx, docker_tool_path, image_name)
java_version = get_java_version(ctx, docker_tool_path, image_name, java_home)
if java_home:
create_java_runtime(ctx, java_home, java_version)
toolchain_config_spec_name = ctx.attr.toolchain_config_spec_name
if ctx.attr.config_version:
# If we found a config we pass it to the toolchain_config_spec_name so when
# we produce platform BUILD file we can use it.
toolchain_config_spec_name = ctx.attr.config_version
else:
# If no config_version was found, generate configs
config_repos = []
if ctx.attr.create_cc_configs:
config_repos.extend(_CONFIG_REPOS)
if ctx.attr.config_repos:
config_repos.extend(ctx.attr.config_repos)
if config_repos:
# run the container and extract the autoconf directory
run_and_extract(
ctx,
bazel_version = ctx.attr.bazel_version,
bazel_rc_version = ctx.attr.bazel_rc_version,
config_repos = config_repos,
docker_tool_path = docker_tool_path,
image_name = image_name,
project_root = mount_project_root,
use_default_project = use_default_project,
)
if ctx.attr.export_configs:
ctx.report_progress("expanding outputs")
# If the user requested exporting configs and did not set a toolchain_config_spec_name lets pick the default
if not toolchain_config_spec_name:
toolchain_config_spec_name = ctx.attr.toolchain_config_suite_spec["default_toolchain_config_spec"]
# Create a default BUILD file with the platform + toolchain targets that
# will work with RBE with the produced toolchain (to be exported to
# output_dir)
ctx.report_progress("creating export platform")
create_export_platform(
ctx,
exec_properties = ctx.attr.internal_exec_properties,
exec_compatible_with = exec_compatible_with,
target_compatible_with = target_compatible_with,
image_name = resolve_rbe_original_image_name(ctx, image_name),
name = name,
toolchain_config_spec_name = toolchain_config_spec_name,
use_legacy_platform_definition = ctx.attr.use_legacy_platform_definition,
)
# Create the versions.bzl file
if ctx.attr.create_versions:
create_versions_file(
ctx,
digest = digest,
toolchain_config_spec_name = toolchain_config_spec_name,
java_home = java_home,
java_version = java_version,
project_root = export_project_root,
)
# Expand outputs to project dir
expand_outputs(
ctx,
bazel_version = ctx.attr.bazel_version,
project_root = export_project_root,
toolchain_config_spec_name = toolchain_config_spec_name,
)
else:
ctx.report_progress("creating external repo platform")
create_external_repo_platform(
ctx,
exec_properties = ctx.attr.internal_exec_properties,
exec_compatible_with = exec_compatible_with,
target_compatible_with = target_compatible_with,
image_name = resolve_rbe_original_image_name(ctx, image_name),
name = name,
use_legacy_platform_definition = ctx.attr.use_legacy_platform_definition,
)
create_configs_tar(ctx)
# If we found checked in confs or if outputs were moved
# to output_base create the alisases.
if ctx.attr.config_version or ctx.attr.export_configs:
create_config_aliases(ctx, toolchain_config_spec_name)
create_alias_platform(
ctx,
exec_properties = ctx.attr.internal_exec_properties,
exec_compatible_with = exec_compatible_with,
target_compatible_with = target_compatible_with,
image_name = resolve_rbe_original_image_name(ctx, image_name),
name = name,
toolchain_config_spec_name = toolchain_config_spec_name,
use_legacy_platform_definition = ctx.attr.use_legacy_platform_definition,
)
# Copy all outputs to the test directory
if ctx.attr.create_testdata:
copy_to_test_dir(ctx)
# Private declaration of _rbe_autoconfig repository rule. Do not use this
# rule directly, use rbe_autoconfig macro declared below.
_rbe_autoconfig = repository_rule(
attrs = {
"base_container_digest": attr.string(
doc = ("Optional. If the container to use for the RBE build " +
"extends from the one defined in the 'toolchain_config_suite_spec' " +
"(defaults to rbe-ubuntu16-04 image), you can " +
"pass the digest (sha256 sum) of the base container here " +
"and this rule will attempt to use checked-in " +
"configs if possible." +
"The digest (sha256 sum) of the base image. " +
"For example, " +
"sha256:87fe00c5c4d0e64ab3830f743e686716f49569dadb49f1b1b09966c1b36e153c" +
", note the digest includes 'sha256:'"),
),
"bazel_rc_version": attr.int(
doc = ("Optional. An rc version to use. Note an installer for " +
"the rc must be available in https://releases.bazel.build."),
),
"bazel_to_config_spec_names_map": attr.string_list_dict(
doc = ("Set by rbe_autoconfig macro. A dict with keys corresponding to bazel versions, " +
"values corresponding to lists of configs. Must point to the " +
"bazel_to_config_versions def in the versions.bzl file " +
"located in the 'output_base' of the 'toolchain_config_suite_spec'."),
),
"bazel_version": attr.string(
default = "local",
doc = ("The version of Bazel to use to generate toolchain configs." +
"Use only (major, minor, patch), e.g., '0.20.0'."),
mandatory = True,
),
"toolchain_config_spec_name": attr.string(
doc = ("The name of the toolchain config spec to be generated."),
),
"configs_obj_config_repos": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'config_repos' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_create_cc_configs": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'cc_configs' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_create_java_configs": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'java_configs' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_env_keys": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'env_keys' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_env_values": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'env_values' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_java_home": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'java_home' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_java_version": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'java_version' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"configs_obj_names": attr.string_list(
doc = ("Set by rbe_autoconfig macro. Set to list 'names' generated by config_to_string_lists def in " +
"//rules/rbe_repo/toolchain_config_suite_spec.bzl."),
),
"config_repos": attr.string_list(
doc = ("Set by rbe_autoconfig macro. list of additional external repos corresponding to " +
"configure like repo rules that need to be produced in addition to " +
"local_config_cc."),
),
"config_version": attr.string(
doc = ("The config version found for the given container and " +
"Bazel version. " +
"Used internally when use_checked_in_confs is true."),
),
"container_to_config_spec_names_map": attr.string_list_dict(
doc = ("Set by rbe_autoconfig macro. A dict with keys corresponding to containers and " +
"values corresponding to lists of configs. Must point to the " +
"container_to_config_version def in the versions.bzl file " +
"located in the 'output_base' of the 'toolchain_config_suite_spec'."),
),
"create_cc_configs": attr.bool(
doc = (
"Specifies whether to generate C/C++ configs. " +
"Defauls to True."
),
mandatory = True,
),
"create_java_configs": attr.bool(
doc = (
"Optional. Specifies whether to generate java configs. " +
"Defauls to True."
),
mandatory = True,
),
"create_testdata": attr.bool(
doc = (
"Specifies whether to generate additional " +
"testing only outputs. " +
"Defauls to False."
),
mandatory = True,
),
# TODO(ngiraldo): remove once migration to use generated file completes
"create_versions": attr.bool(
doc = (
"Specifies whether to generate versions.bzl " +
"file in output_base of the toolchain_config_suite_spec. " +
"This option is temporary while migration to use. " +
"generated file by this rule is taking place. " +
"Defauls to True."
),
mandatory = True,
),
"detect_java_home": attr.bool(
doc = (
"Specifies whether to find the JAVA_HOME as set in the" +
"container. " +
"Defauls to False."
),
mandatory = True,
),
"digest": attr.string(
doc = ("Optional. The digest (sha256 sum) of the image to pull. " +
"For example, " +
"sha256:87fe00c5c4d0e64ab3830f743e686716f49569dadb49f1b1b09966c1b36e153c" +
", note the digest includes 'sha256:'"),
),
"env": attr.string_dict(
doc = ("Optional. Dictionary from strings to strings. Additional env " +
"variables that will be set when running the Bazel command to " +
"generate the toolchain configs."),
),
"exec_compatible_with": attr.string_list(
doc = ("Optional. The list of constraints that will be added to the " +
"toolchain in its exec_compatible_with attribute (and to " +
"the platform in its constraint_values attr). For " +
"example, [\"@bazel_tools//platforms:linux\"]."),
),
"internal_exec_properties": attr.string_dict(
doc = (
"Optional. The execution properties to be used when creating the " +
"underlying platform. When providing this attribute, " +
"use_legacy_platform_definition must be set to False. Note that " +
"the container image property must not be specified via this " +
"attribute."
),
),
"export_configs": attr.bool(
doc = (
"Specifies whether to copy generated configs to the 'output_base' " +
"of the 'toolchain_config_suite_spec' (if configs are generated) " +
"If set to False, a configs.tar file will also be produced in the " +
("external repo. This tar file can be then published to a URL and " +
" e.g., be used via an 'http_archive' rule from an arbitrary repo." +
"Default is False.")
),
mandatory = True,
),
"java_home": attr.string(
doc = ("Optional. The location of java_home in the container. For " +
"example , '/usr/lib/jvm/java-8-openjdk-amd64'. Only " +
"relevant if 'create_java_configs' is true. If 'create_java_configs' is " +
"true, the execution of the rule generates configs, and this attribute " +
"is not set, the rule will attempt to read the " +
"JAVA_HOME env var from the container. If that is not set, the rule " +
"will fail."),
),
"java_version": attr.string(
doc = ("Optional. The Java release version in the container. For " +
" example, 11. Should only be set if java_home is set."),
),
"registry": attr.string(
doc = ("Optional. The registry to pull the container from. For example, " +
"marketplace.gcr.io. The default is the value for the selected " +
"toolchain_config_suite_spec (rbe-ubuntu16-04 image for " +
"default_toolchain_config_suite_spec, if no toolchain_config_suite_spec was selected)."),
),
"repository": attr.string(
doc = ("Optional. The repository to pull the container from. For example, " +
"google/ubuntu. The default is the " +
"value for the selected toolchain_config_suite_spec (rbe-ubuntu16-04 image for " +
"default_toolchain_config_suite_spec, if no toolchain_config_suite_spec was selected)."),
),
"toolchain_config_suite_spec": attr.string_dict(
doc = ("Set by rbe_autoconfig macro. Dict containing values to identify a " +
"toolchain container + GitHub repo where configs are " +
"stored. Must include keys: 'repo_name' (name of the " +
"external repo, 'output_base' (relative location of " +
"the output base in the GitHub repo where configs are " +
"located), and 'container_repo', 'container_registry', " +
"'container_name' (describing the location of the " +
"base toolchain container)"),
allow_empty = False,
mandatory = True,
),
"setup_cmd": attr.string(
default = "cd .",
doc = ("Optional. Pass an additional command that will be executed " +
"(inside the container) before running bazel to generate the " +
"toolchain configs"),
),
"tag": attr.string(
doc = ("Optional. The tag of the image to pull, e.g. latest."),
),
"target_compatible_with": attr.string_list(
doc = ("The list of constraints that will be added to the " +
"toolchain in its target_compatible_with attribute. For " +
"example, [\"@bazel_tools//platforms:linux\"]."),
),
"use_checked_in_confs": attr.string(
default = CHECKED_IN_CONFS_TRY,
doc = ("Default: 'Try'. Try to look for checked in configs " +
"before generating them. If set to 'False' (string) the " +
"rule will allways attempt to generate the configs " +
"by pulling a toolchain container and running Bazel inside. " +
"If set to 'Force' rule will error out if no checked-in" +
"configs were found."),
values = CHECKED_IN_CONFS_VALUES,
),
"use_legacy_platform_definition": attr.bool(
doc = (
"Specifies whether the underlying platform uses the " +
"remote_execution_properties property (if use_legacy_platform_definition " +
"is True) or the exec_properties property. The reason why this " +
"is important is because a platform that inherits from this " +
"platform and wishes to add execution properties must use the " +
"same field remote_execution_properties/exec_properties that " +
"the parent platform uses. This attribute must be set to False if the " +
"exec_properties attribute is set."
),
mandatory = True,
),
},
environ = [
AUTOCONF_ROOT,
DOCKER_PATH,
],
implementation = _rbe_autoconfig_impl,
local = True,
)
def rbe_autoconfig(
name,
base_container_digest = None,
bazel_version = None,
bazel_rc_version = None,
toolchain_config_spec_name = None,
config_repos = None,
create_cc_configs = True,
create_java_configs = True,
create_testdata = False,
create_versions = True,
detect_java_home = False,
digest = None,
env = None,
exec_compatible_with = None,
exec_properties = None,
export_configs = False,
java_home = None,
java_version = None,
tag = None,
toolchain_config_suite_spec = default_toolchain_config_suite_spec(),
registry = None,
repository = None,
target_compatible_with = None,
use_checked_in_confs = CHECKED_IN_CONFS_TRY,
use_legacy_platform_definition = True):
""" Creates a repository with toolchain configs generated for a container image.
This macro wraps (and simplifies) invocation of _rbe_autoconfig rule.
Use this macro in your WORKSPACE.
Args:
name: Name of the rbe_autoconfig repository target.
base_container_digest: Optional. If the container to use for the RBE build
extends from the container defined in the toolchain_config_suite_spec
(by default, the rbe-ubuntu16-04 image), you can pass the digest
(sha256 sum) of the base container using this attr.
The rule will try to use of checked-in configs, if possible.
bazel_version: The version of Bazel to use to generate toolchain configs.
`Use only (major, minor, patch), e.g., '0.20.0'. Default is "local"
which means the same version of Bazel that is currently running will
be used. If local is a non release version, rbe_autoconfig will fallback
to using the latest release version (see _BAZEL_VERSION_FALLBACK).
Note, if configs are not found for a patch version, rule will attempt
to find ones for the corresponding x.x.0 version. So if you are using
Bazel 0.25.2, and configs are not found for that version, but are
available for 0.25.0, those will be used instead. Note: this is only
the case if use_checked_in_confs != "False" (string 'False').
bazel_rc_version: The rc (for the given version of Bazel) to use.
Must be published in https://releases.bazel.build. E.g. 2.
toolchain_config_spec_name: Optional. String. Override default config
defined in toolchain_config_suite_spec.
If export_configs is True, this value is used to set the name of the
toolchain config spec to be generated.
config_repos: Optional. List of additional external repos corresponding to
configure like repo rules that need to be produced in addition to
local_config_cc.
create_cc_configs: Optional. Specifies whether to generate C/C++ configs.
Defauls to True.
create_java_configs: Optional. Specifies whether to generate java configs.
Defauls to True.
create_testdata: Optional. Specifies whether to generate additional testing
only outputs. Defauls to False.
create_versions: Specifies whether to generate versions.bzl
file in 'output_base' of the 'toolchain_config_suite_spec'.
This option is temporary while migration to use.
generated file by this rule is taking place.
Defauls to True.
digest: Optional. The digest of the image to pull.
Should not be set if 'tag' is used.
Must be set together with 'registry' and 'repository'.
detect_java_home: Optional. Default False. Should only be set
to True if 'create_java_configs' is also True. If set to True the rule
will attempt to read the JAVA_HOME env var from the container.
Note if java_home is not set and this is set to False, the rule will
attempt to find a value of java_home in a compatible
'toolchain_config_spec', fallback to using the 'default_java_home' in
the 'toolchain_config_suite_spec', fallback to turning on 'detect_java_home,
(unless use_checked_in_confs = Force was set), or otherwise fail with an
informative error.
env: dict. Optional. Additional environment variables that will be set when
running the Bazel command to generate the toolchain configs.
Set to values for marketplace.gcr.io/google/rbe-ubuntu16-04 container.
Note: Do not pass a custom JAVA_HOME via env, use java_home attr instead.
exec_compatible_with: Optional. List of constraints to add to the produced
toolchain/platform targets (e.g., ["@bazel_tools//platforms:linux"] in the
exec_compatible_with/constraint_values attrs, respectively.
exec_properties: Optional. A string->string dict containing execution
properties to be used when creating the underlying platform. When
providing this attribute use_legacy_platform_definition must be set
to False. Note that the container image property must not be specified
via this attribute.
export_configs: Optional, default False. Whether to copy generated configs
(if they are generated) to the 'output_base' defined in
'toolchain_config_suite_spec'. If set to False, a configs.tar file
will also be produced in the external repo. This tar file can be then
published to a URL and e.g., be used via an 'http_archive' rule
from an arbitrary repo.
java_home: Optional. The location of java_home in the container. For
example , '/usr/lib/jvm/java-8-openjdk-amd64'. Should only be set
if 'create_java_configs' is True. Cannot be set if detect_java_home
is set to True.
Note if detect_java_home is set to False and this is not set, the
rule will attempt to find a value of java_home in a compatible
'toolchain_config_spec', fallback to using the 'default_java_home' in