[feature] Add group templates openwisp#631

Implements and closes openwisp#631

Co-authored-by: Federico Capoano <f.capoano@openwisp.io>

README.rst

@@ -919,18 +919,58 @@ registered with the same name.
 Device Groups
 ~~~~~~~~~~~~~
 
-Device Groups provide an easy way to organize devices of a particular organization.
-Device Groups provide the following features:
+Device Groups provide features aimed at adding specific management rules
+for the devices of an organization:
 
 - Group similar devices by having dedicated groups for access points, routers, etc.
-- Store additional information regarding a group in the structured metadata field.
+- Store additional information regarding a group in the structured metadata field
+  (which can be accessed via the REST API).
 - Customize structure and validation of metadata field of DeviceGroup to standardize
   information across all groups using `"OPENWISP_CONTROLLER_DEVICE_GROUP_SCHEMA" <#openwisp-controller-device-group-schema>`_
   setting.
+- Define `group configuration templates <#group-templates>`_.
 
-.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/device-groups.png
+.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.1/device-groups.png
   :alt: Device Group example
 
+Group Templates
+###############
+
+Groups allow to define templates which are automatically assigned to devices
+belonging to the group. When using this feature, keep in mind the following
+important points:
+
+- Templates of any configuration backend can be selected,
+  when a device is assigned to a group,
+  only the templates which matches the device configuration backend are
+  applied to the device.
+- The system will not force group templates onto devices, this means that
+  users can remove the applied group templates from a specific device if
+  needed.
+- If a device group is changed, the system will automatically remove the
+  group templates of the old group and apply the new templates of the new
+  group (this operation is implemented by leveraging the
+  `group_templates_changed <#group_templates_changed>`_ signal).
+- If the group templates are changed, the devices which belong to the group
+  will be automatically updated to reflect the changes
+  (this operation is executed in a background task).
+- In case the configuration backend of a device is changed,
+  the system will handle this automatically too and update the group
+  templates accordingly (this operation is implemented by leveraging the
+  `config_backend_changed <#config_backend_changed>`_ signal).
+- If a device does not have a configuration defined yet, but it is assigned
+  to a group which has templates defined, the system will automatically
+  create a configuration for it using the default backend specified in
+  `OPENWISP_CONTROLLER_DEFAULT_BACKEND <#OPENWISP_CONTROLLER_DEFAULT_BACKEND>`_ setting.
+
+**Note:** the list of templates shown in the edit group page do not
+contain templates flagged as "default" or "required" to avoid redundancy
+because those templates are automatically assigned by the system
+to new devices.
+
+This feature works also when editing group templates or the group assigned
+to a device via the `REST API <#change-device-group-detail>`__.
+
 Export/Import Device data
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -1594,6 +1634,15 @@ Get device group detail
 
     GET /api/v1/controller/group/{id}/
 
+Change device group detail
+##########################
+
+.. code-block:: text
+
+    PUT /api/v1/controller/group/{id}/
+
+This endpoint allows to change the `group templates <#group-templates>`_ too.
+
 Get device group from certificate common name
 #############################################
 
@@ -2926,6 +2975,18 @@ object are changed, but only on ``post_add`` or ``post_remove`` actions,
 ``post_clear`` is ignored for the same reason explained
 in the previous section.
 
+``config_backend_changed``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Path**: ``openwisp_controller.config.signals.config_backend_changed``
+**Arguments**:
+
+- ``instance``: instance of ``Config`` which got its ``backend`` changed
+- ``old_backend``: the old backend of the config object
+- ``backend``: the new backend of the config object
+
+It is not emitted when the device or config is created.
+
 ``checksum_requested``
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -3035,6 +3096,22 @@ The signal is emitted when the device group changes.
 
 It is not emitted when the device is created.
 
+``group_templates_changed``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+**Path**: ``openwisp_controller.config.signals.group_templates_changed``
+
+**Arguments**:
+
+- ``instance``: instance of ``DeviceGroup``.
+- ``templates``: list of ``Template`` objects assigned to ``DeviceGroup``
+- ``old_templates``: list of ``Template`` objects assigned earlier to ``DeviceGroup``
+
+The signal is emitted when the device group templates changes.
+
+It is not emitted when the device is created.
+
 ``subnet_provisioned``
 ~~~~~~~~~~~~~~~~~~~~~~
 

openwisp_controller/config/admin.py

@@ -317,6 +317,8 @@ class Meta:
 
 
 class ConfigForm(AlwaysHasChangedMixin, BaseForm):
+    _old_templates = None
+
     def get_temp_model_instance(self, **options):
         config_model = self.Meta.model
         instance = config_model(**options)
@@ -357,8 +359,24 @@ def clean_templates(self):
                 # the device object.
                 raw_data=self.data,
             )
+        self._old_templates = list(config.templates.filter(required=False))
         return templates
 
+    def save(self, *args, **kwargs):
+        templates = self.cleaned_data.pop('templates', [])
+        instance = super().save(*args, **kwargs)
+        # as group templates are not forced so if user remove any selected
+        # group template, we need to remove it from the config instance
+        # not doing this in save_m2m because save_form_data directly set the
+        # user selected templates and we need to handle the condition i.e.
+        # group templates get applied at the time of creation of config
+        instance.manage_group_templates(
+            templates=templates,
+            old_templates=self._old_templates,
+            ignore_backend_filter=True,
+        )
+        return instance
+
     class Meta(BaseForm.Meta):
         model = Config
         widgets = {'config': JsonSchemaWidget, 'context': FlatJsonWidget}
@@ -461,9 +479,10 @@ class DeviceAdmin(MultitenantAdminMixin, BaseConfigAdmin, UUIDAdmin):
     inlines = [ConfigInline]
     conditional_inlines = []
     actions = ['change_group']
-
     org_position = 1 if not app_settings.HARDWARE_ID_ENABLED else 2
     list_display.insert(org_position, 'organization')
+    _state_adding = False
+    _config_formset = None
 
     if app_settings.CONFIG_BACKEND_FIELD_SHOWN:
         list_filter.insert(1, 'config__backend')
@@ -480,12 +499,65 @@ class Media(BaseConfigAdmin.Media):
             f'{prefix}js/relevant_templates.js',
         ]
 
+    def save_form(self, request, form, change):
+        self._state_adding = form.instance._state.adding
+        return super().save_form(request, form, change)
+
+    def save_formset(self, request, form, formset, change):
+        # if a new device and config objects get created
+        # with a group having group templates assigned,
+        # the device group functionality creates a
+        # new config for the device before this form
+        # is saved, therefore we'll incur in an integrity
+        # error exception because the config already exists.
+        # To avoid that, we have to convince django that
+        # the formset is for an existing object and not a new one
+        if (
+            self._state_adding
+            and formset.model == Config
+            and hasattr(form.instance, 'config')
+            and form.instance.group
+            and form.instance.group.templates.exists()
+        ):
+            formset.data['config-0-id'] = str(form.instance.config.id)
+            formset.data['config-0-device'] = str(form.instance.id)
+            formset.data['config-INITIAL_FORMS'] = '1'
+            templates = form.instance.config.templates.all().values_list(
+                'pk', flat=True
+            )
+            templates = [str(template) for template in templates]
+            formset.data['config-0-templates'] = ','.join(templates)
+            formset_new = formset.__class__(
+                data=formset.data, instance=formset.instance
+            )
+            formset = formset_new
+            formset.full_clean()
+            formset.new_objects = []
+            formset.changed_objects = []
+            formset.deleted_objects = []
+            self._config_formset = formset
+        return super().save_formset(request, form, formset, change)
+
+    def construct_change_message(self, request, form, formsets, add=False):
+        if self._state_adding and self._config_formset:
+            formsets[0] = self._config_formset
+        return super().construct_change_message(request, form, formsets, add)
+
     def change_group(self, request, queryset):
         if 'apply' in request.POST:
             form = ChangeDeviceGroupForm(request.POST)
             if form.is_valid():
                 group = form.cleaned_data['device_group']
+                instances, old_group_ids = map(
+                    list, zip(*queryset.values_list('id', 'group'))
+                )
                 queryset.update(group=group or None)
+                group_id = None
+                if group:
+                    group_id = group.id
+                Device._send_device_group_changed_signal(
+                    instance=instances, group_id=group_id, old_group_id=old_group_ids
+                )
             self.message_user(
                 request,
                 _('Successfully changed group of selected devices.'),
@@ -849,6 +921,24 @@ class Media(BaseConfigAdmin):
 
 
 class DeviceGroupForm(BaseForm):
+    _templates = None
+
+    def clean_templates(self):
+        templates = self.cleaned_data.get('templates')
+        self._templates = [template.id for template in templates]
+        return templates
+
+    def save(self, *args, **kwargs):
+        instance = super().save(*args, **kwargs)
+        old_templates = list(self.instance.templates.values_list('pk', flat=True))
+        if not self.instance._state.adding and old_templates != self._templates:
+            DeviceGroup.templates_changed(
+                instance=instance,
+                old_templates=old_templates,
+                templates=self._templates,
+            )
+        return instance
+
     class Meta(BaseForm.Meta):
         model = DeviceGroup
         widgets = {'meta_data': DeviceGroupJsonSchemaWidget}
@@ -878,6 +968,7 @@ def queryset(self, request, queryset):
 
 
 class DeviceGroupAdmin(MultitenantAdminMixin, BaseAdmin):
+    change_form_template = 'admin/device_group/change_form.html'
     form = DeviceGroupForm
     list_display = [
         'name',
@@ -889,14 +980,19 @@ class DeviceGroupAdmin(MultitenantAdminMixin, BaseAdmin):
         'name',
         'organization',
         'description',
+        'templates',
         'meta_data',
         'created',
         'modified',
     ]
     search_fields = ['name', 'description', 'meta_data']
     list_filter = [('organization', MultitenantOrgFilter), DeviceGroupFilter]
+    multitenant_shared_relations = ('templates',)
 
     class Media:
+        js = list(UUIDAdmin.Media.js) + [
+            f'{prefix}js/relevant_templates.js',
+        ]
         css = {'all': (f'{prefix}css/admin.css',)}
 
     def get_urls(self):
@@ -915,6 +1011,23 @@ def get_urls(self):
     def schema_view(self, request):
         return JsonResponse(app_settings.DEVICE_GROUP_SCHEMA)
 
+    def add_view(self, request, form_url='', extra_context=None):
+        extra_context = extra_context or {}
+        extra_context.update(self.get_extra_context())
+        return super().add_view(request, form_url, extra_context)
+
+    def change_view(self, request, object_id, form_url='', extra_context=None):
+        extra_context = self.get_extra_context(object_id)
+        return super().change_view(request, object_id, form_url, extra_context)
+
+    def get_extra_context(self, pk=None):
+        ctx = {
+            'relevant_template_url': reverse(
+                'admin:get_relevant_templates', args=['org_id']
+            ),
+        }
+        return ctx
+
 
 admin.site.register(Device, DeviceAdminExportable)
 admin.site.register(Template, TemplateAdmin)

openwisp_controller/config/api/serializers.py

@@ -162,7 +162,11 @@ def create(self, validated_data):
         with transaction.atomic():
             device = Device.objects.create(**validated_data)
             if config_data:
-                config = Config.objects.create(device=device, **config_data)
+                if not hasattr(device, 'config'):
+                    config = Config.objects.create(device=device, **config_data)
+                else:
+                    Config.objects.filter(device=device).update(**config_data)
+                    config = device.config
                 config.templates.add(*config_templates)
         return device
 
@@ -276,8 +280,15 @@ def update(self, instance, validated_data):
         return super().update(instance, validated_data)
 
 
+class FilterGroupTemplates(FilterTemplatesByOrganization):
+    def get_queryset(self):
+        return super().get_queryset().exclude(Q(default=True) | Q(required=True))
+
+
 class DeviceGroupSerializer(BaseSerializer):
     meta_data = serializers.JSONField(required=False, initial={})
+    templates = FilterGroupTemplates(many=True)
+    _templates = None
 
     class Meta(BaseMeta):
         model = DeviceGroup
@@ -286,7 +297,33 @@ class Meta(BaseMeta):
             'name',
             'organization',
             'description',
+            'templates',
             'meta_data',
             'created',
             'modified',
         ]
+
+    def validate(self, data):
+        self._templates = [template.id for template in data.pop('templates', [])]
+        return super().validate(data)
+
+    def _save_m2m_templates(self, instance, created=False):
+        old_templates = list(instance.templates.values_list('pk', flat=True))
+        if old_templates != self._templates:
+            instance.templates.set(self._templates)
+            if not created:
+                self.Meta.model.templates_changed(
+                    instance=instance,
+                    old_templates=old_templates,
+                    templates=self._templates,
+                )
+
+    def create(self, validated_data):
+        instance = super().create(validated_data)
+        self._save_m2m_templates(instance, created=True)
+        return instance
+
+    def update(self, instance, validated_data):
+        instance = super().update(instance, validated_data)
+        self._save_m2m_templates(instance)
+        return instance