From 54bca200e51a8c9ed848f3ab6ba673d060d55bfe Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 19 Dec 2024 19:17:14 +0100 Subject: [PATCH 01/17] docs: add create a new filter documents --- ...new-filter.rst => create-a-new-filter.rst} | 0 docs/how-tos/index.rst | 3 +- docs/reference/architeture-subdomains.rst | 31 +++++++++++++++++++ docs/reference/filters-configuration.rst | 0 4 files changed, 32 insertions(+), 2 deletions(-) rename docs/how-tos/{create-new-filter.rst => create-a-new-filter.rst} (100%) create mode 100644 docs/reference/architeture-subdomains.rst create mode 100644 docs/reference/filters-configuration.rst diff --git a/docs/how-tos/create-new-filter.rst b/docs/how-tos/create-a-new-filter.rst similarity index 100% rename from docs/how-tos/create-new-filter.rst rename to docs/how-tos/create-a-new-filter.rst diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 836fad40..2325ccf8 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -5,5 +5,4 @@ How-tos :maxdepth: 1 :caption: Contents: - create-new-filter - using-filters + create-a-new-filter diff --git a/docs/reference/architeture-subdomains.rst b/docs/reference/architeture-subdomains.rst new file mode 100644 index 00000000..c65eb991 --- /dev/null +++ b/docs/reference/architeture-subdomains.rst @@ -0,0 +1,31 @@ +Architecture Subdomains +======================= + +Currently, these are the `architecture subdomains`_ used by the Open edX Events library: + ++-------------------+----------------------------------------------------------------------------------------------------+ +| Subdomain name | Description | ++===================+====================================================================================================+ +| Content Authoring | Allows educators to create, modify, package, annotate (tag), and share learning content. | ++-------------------+----------------------------------------------------------------------------------------------------+ +| Learning | Allows learners to consume content and perform actions in a learning activity on the platform. | ++-------------------+----------------------------------------------------------------------------------------------------+ +| Analytics | Provides insights into learner behavior and course performance. | ++-------------------+----------------------------------------------------------------------------------------------------+ +| Enterprise | Provides tools for organizations to manage their learners and courses. | ++-------------------+----------------------------------------------------------------------------------------------------+ + +Here we list useful information about Open edX architecture subdomains and their use in the Hooks Extension framework: + +- `Events Naming and Versioning`_ +- `Notes on events design and subdomains`_ +- `edX Domain Driven Design documentation`_ +- `Subdomains from OEP-41`_ +- `Message Content Data Guidelines`_ + +.. _Events Naming and Versioning: https://github.com/openedx/openedx-events/blob/main/docs/decisions/0002-events-naming-and-versioning.rst#L1 +.. _edX Domain Driven Design documentation: https://openedx.atlassian.net/wiki/spaces/AC/pages/213910332/Domain-Driven+Design +.. _`Subdomains from OEP-41`: https://docs.openedx.org/projects/openedx-proposals/en/latest/architectural-decisions/oep-0041-arch-async-server-event-messaging.html#subdomain-from-domain-driven-design +.. _`Message Content Data Guidelines`: https://docs.openedx.org/projects/openedx-proposals/en/latest/architectural-decisions/oep-0041-arch-async-server-event-messaging.html?highlight=subdomain#message-content-data-guidelines +.. _`Notes on events design and subdomains`: https://github.com/openedx/openedx-events/issues/72#issuecomment-1179291340 +.. _architecture subdomains: https://microservices.io/patterns/decomposition/decompose-by-subdomain.html diff --git a/docs/reference/filters-configuration.rst b/docs/reference/filters-configuration.rst new file mode 100644 index 00000000..e69de29b From 6f3c3fdbd21e8d15479b1e3720e0b93288e2e4ce Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 19 Dec 2024 19:19:04 +0100 Subject: [PATCH 02/17] docs: drop using filter in favor of create a new filter --- docs/how-tos/using-filters.rst | 157 --------------------------------- 1 file changed, 157 deletions(-) delete mode 100644 docs/how-tos/using-filters.rst diff --git a/docs/how-tos/using-filters.rst b/docs/how-tos/using-filters.rst deleted file mode 100644 index 92adba67..00000000 --- a/docs/how-tos/using-filters.rst +++ /dev/null @@ -1,157 +0,0 @@ -How to use Open edX Filters ---------------------------- - -Using openedx-filters in your code is very straight forward. We can consider the -various use cases: implementing :term:`Pipeline Steps`, attaching/hooking pipelines to filter, -and triggering a filter. We'll also cover how to test the filters you create in your service. - - -Implement pipeline steps -************************ - -Let's say you want to consult student's information with a third party service -before generating the students certificate. This is a common use case for filters, -where the functions part of the :term:`filter pipeline` will perform the consulting tasks and -decide the execution flow for the application. These functions are the :term:`pipeline steps`, -and can be implemented in an installable Python library: - -.. code-block:: python - - # Step implementation taken from openedx-filters-samples plugin - from openedx_filters import PipelineStep - from openedx_filters.learning.filters import CertificateCreationRequested - - class StopCertificateCreation(PipelineStep): - """ - Stop certificate creation if user is not in third party service. - """ - - def run_filter(self, user, course_id, mode, status): - # Consult third party service and check if continue - # ... - # User not in third party service, denied certificate generation - raise CertificateCreationRequested.PreventCertificateCreation( - "You can't generate a certificate from this site." - ) - -There's two key components to the implementation: - -1. The filter step must be a subclass of ``PipelineStep``. - -2. The ``run_filter`` signature must match the filters', eg., the step signature matches the `run_filter` signature in CertificateCreationRequested: - -.. code-block:: python - - class CertificateCreationRequested(OpenEdxPublicFilter): - """ - Custom class used to create certificate creation filters and its custom methods. - """ - - filter_type = "org.openedx.learning.certificate.creation.requested.v1" - - class PreventCertificateCreation(OpenEdxFilterException): - """ - Custom class used to stop the certificate creation process. - """ - - @classmethod - def run_filter(cls, user, course_key, mode, status, grade, generation_mode): - """ - Execute a filter with the signature specified. - - Arguments: - user (User): is a Django User object. - course_key (CourseKey): course key associated with the certificate. - mode (str): mode of the certificate. - status (str): status of the certificate. - grade (CourseGrade): user's grade in this course run. - generation_mode (str): Options are "self" (implying the user generated the cert themself) and "batch" - for everything else. - """ - data = super().run_pipeline( - user=user, course_key=course_key, mode=mode, status=status, grade=grade, generation_mode=generation_mode, - ) - return ( - data.get("user"), - data.get("course_key"), - data.get("mode"), - data.get("status"), - data.get("grade"), - data.get("generation_mode"), - ) - -Attach/hook pipeline to filter -****************************** - -After implementing the :term:`pipeline steps`, we have to tell the certificate creation -filter to execute our :term:`pipeline`. - -.. code-block:: python - - OPEN_EDX_FILTERS_CONFIG = { - "org.openedx.learning.certificate.creation.requested.v1": { - "fail_silently": False, - "pipeline": [ - "openedx_filters_samples.samples.pipeline.StopCertificateCreation" - ] - }, - } - -Triggering a filter -******************* - -In order to execute a filter in edx-platform or your own plugin/library, you must install the -plugin where the steps are implemented and also, ``openedx-filters``. - -.. code-block:: python - - # Code taken from lms/djangoapps/certificates/generation_handler.py - from openedx_filters.learning.filters import CertificateCreationRequested - - try: - user, course_id, mode, status = CertificateCreationRequested.run_filter( - user=user, course_id=course_id, mode=mode, status=status, - ) - except CertificateCreationRequested.PreventCertificateCreation as exc: - raise CertificateGenerationNotAllowed(str(exc)) from exc - -Testing filters' steps -********************** - -It's pretty straightforward to test your pipeline steps, you'll need to include the -``openedx-filters`` library in your testing dependencies and configure them in your test case. - -.. code-block:: python - - from openedx_filters.learning.filters import CertificateCreationRequested - - @override_settings( - OPEN_EDX_FILTERS_CONFIG={ - "org.openedx.learning.certificate.creation.requested.v1": { - "fail_silently": False, - "pipeline": [ - "openedx_filters_samples.samples.pipeline.StopCertificateCreation" - ] - } - } - ) - def test_certificate_creation_requested_filter(self): - """ - Test filter triggered before the certificate creation process starts. - - Expected results: - - The pipeline step configured for the filter raises PreventCertificateCreation - when the conditions are met. - """ - ... - with self.assertRaises(CertificateCreationRequested.PreventCertificateCreation): - CertificateCreationRequested.run_filter( - user=user, course_key=course_key, mode="audit", - ) - - # run your assertions - -Changes in the ``openedx-filters`` library that are not compatible with your code -should break this kind of test in CI and let you know you need to upgrade your code. -The main limitation while testing filters' steps is their arguments, as they are -in-memory objects, but that can be solved in CI using Python mocks. From 07210987157e639684b81dae910f5665703d23d4 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Fri, 20 Dec 2024 15:22:31 +0100 Subject: [PATCH 03/17] docs: update documentation with latest how-to steps --- docs/Makefile | 3 + docs/how-tos/create-a-new-filter.rst | 268 ++++++++---------- ...omains.rst => architecture-subdomains.rst} | 0 docs/reference/filters-configuration.rst | 49 ++++ docs/reference/glossary.rst | 2 +- docs/reference/index.rst | 2 + 6 files changed, 178 insertions(+), 146 deletions(-) rename docs/reference/{architeture-subdomains.rst => architecture-subdomains.rst} (100%) diff --git a/docs/Makefile b/docs/Makefile index d4bb2cbb..115749bf 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -18,3 +18,6 @@ help: # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +serve_docs: ## serve the built docs locally to preview the site in the browser + sphinx-autobuild . $(BUILDDIR)/html diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index 0ab554c5..9097fce8 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -1,205 +1,183 @@ -How-to Create a new Filter -########################## +Create a New Open edX Filter with Long-Term Support +#################################################### -.. How-tos should have a short introduction sentence that captures the user's goal and introduces the steps. +Open edX Filters are supported and maintained by the Open edX community. This mechanism is designed to be extensible and flexible to allow developers to create new filters to implement custom behavior in the application. This guide describes how to create a new Open edX filter with long-term support by following the practices outlined in the :doc:`../decisions/0007-filter-design-practices` ADR. -The mechanisms implemented by the Open edX Filters library are supported and maintained by the Open edX community. The -library is designed to be extensible, and we welcome contributions of new filters. +Filters design with long-support follow closely the practices described in the ADR to minimize breaking changes, maximize compatibility and support for future versions of Open edX. -Therefore, we've put together this guide that will walk you through the process of adding a new filter to the library, -and will provide you with a template to follow when adding new filters. +.. note:: Before starting, ensure you've reviewed the documentation on :doc:`docs.openedx.org:developers/concepts/hooks_extension_framework`, this documentation helps you decide if creating a new filter is necessary. You should also review the documentation on :doc:`../decisions/0007-filter-design-practices` to understand the practices that should be followed when creating a new filter. -Assumptions -*********** +Throughout this guide, we will use an example of creating a new filter that will be triggered when a user enrolls in a course from the course about page to better illustrate the steps involved in creating a new filter. + +Key Outlines from Filter Design Practices +----------------------------------------- -.. This section should contain a bulleted list of assumptions you have of the - person who is following the How-to. The assumptions may link to other - how-tos if possible. +- Clearly describe the behavior the filter modifies. +- Use concise names that reflect the filter's purpose. +- Ensure consistent and narrow triggering logic. +- Provide sufficient context in arguments to modify intended behavior. +- Avoid runtime dependencies by including relevant context in arguments. +- Keep arguments closely tied to the filter's responsibility. +- Allow flexibility for developers to customize behavior. +- Handle exceptions properly to halt the application behavior when needed without breaking the application. +- Align exceptions with filter behavior and specify when halting is needed. +- Annotate the argument types for clarity and safety. + +Assumptions +----------- -* You have a development environment set up. -* You have a basic understanding of Python and Django. -* You understand the concept of filters or have reviewed the relevant - :doc:`/concepts/index` docs. -* You are familiar with the terminology used in the project, such as - :term:`filter type`, :term:`filter signature`, etc. If not, you can review the :doc:`/reference/glossary` docs. +- You have a development environment set up using `Tutor`_. +- You have a basic understanding of Python and Django. +- You understand the concept of filters or have reviewed the relevant :doc:`/concepts/index` docs. +- You are familiar with the terminology used in the project, such as the terms :term:`Filter Type`. If not, you can review the :doc:`../reference/glossary` docs. +- You have reviewed the :doc:`../decisions/0007-filter-design-practices` ADR. +- You have identified that you need to create a new filter and have a use case for the filter. Steps -***** +----- -.. A task should have 3 - 7 steps. Tasks with more should be broken down into digestible chunks. +To create a new Open edX Filter with long-term support, follow these steps: -#. Propose a new filter to the Open edX community +Step 1: Propose the Use Case to the Community +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - When creating a new filter, you must justify its implementation. For example, you could create a post in Discuss, - send a message through slack or open a new issue in the library repository listing your use cases for it. Or even, - if you have time, you could accompany your proposal with the implementation of the filter to illustrate its behavior. +Before contributing a new filter, it is important to propose the filter to the community to get feedback on the filter's design and use case. For instance, you could create a post in Open edX Discuss Forum or create a new issue in the repository's issue tracker describing your use case for the new filter. Here is an example of community members that have taken this step: -#. Place your filter in an architecture subdomain +- `Add Extensibility Mechanism to IDV to Enable Integration of New IDV Vendor Persona`_ - As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the :term:`filter definition` needs an Open edX Architecture - Subdomain for: +.. note:: If your use case is too specific to your organization, you can implement them in your own library and use it within your services by adopting an organization-scoped approach. However, if you think that your use case could be beneficial to the community, you should propose it to the community for feedback and collaboration. - - The :term:`type of the filter`: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` - - The package name where the definition will live, eg. ``learning/``. +In our example our use case proposal could be: - For those reasons, after studying your new filter purpose, you must place it in one of the subdomains already in use, or introduce a new subdomain: + I want to add a filter that will be triggered when a user enrolls in a course from the course about page. This filter will be used to prfilter users from enrolling in a course if they do not meet the eligibility criteria. The filter will be triggered when the user clicks the enroll button on the course about page and will check if the user meets the eligibility criteria. If the user does not meet the criteria, the filter will raise an exception to prfilter the user from enrolling in the course. - +-------------------+----------------------------------------------------------------------------------------------------+ - | Subdomain name | Description | - +===================+====================================================================================================+ - | Learning | Allows learners to consume content and perform actions in a learning activity on the platform. | - +-------------------+----------------------------------------------------------------------------------------------------+ +If you are confident that the filter is beneficial to the community, you can proceed to the next steps and implement the filter. - New subdomains may require some discussion, because there does not yet exist and agreed upon set of subdomains. So we encourage you to start the conversation - as soon as possible through any of the communication channels available. +Step 2: Place Your Filter in an Architecture Subdomain +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Refer to `edX DDD Bounded Contexts `_ confluence page for more documentation on domain-driven design in the Open edX project. +To implement the new filter in the library, you should understand the purpose of the filter and where it fits in the Open edX main architecture subdomains. This will help you place the filter in the right architecture subdomain and ensure that the filter is consistent with the framework's definitions. Fore more details on the Open edX Architectural Subdomains, refer to the :doc:`../reference/architecture-subdomains`. -#. Define the filter's behavior +In our example, the filter is related to the enrollment process, which is part of the ``learning`` subdomain. Therefore, the filter should be placed in the ``/learning`` module in the library. The subdomain is also used as part of the :term:`filter type `, which is used to identify the filter. The filter type should be unique and follow the naming convention for filter types specified in the :doc:`../decisions/0004-filters-naming-and-versioning` ADR. - Defining the filter's behavior includes: +For the enrollment filter, the filter type could be ``org.openedx.learning.course.enrollment.v1``, where ``learning`` is the subdomain. - - Defining the :term:`filter type` for identification - - Defining the :term:`filter signature` - - Defining the filter's behavior for stopping the process in which it is being used +.. note:: If you don't find a suitable subdomain for your filter, you can propose a new subdomain to the community. However, new subdomains may require some discussion with the community. So we encourage you to start the conversation as soon as possible through any of the communication channels available. - The :term:`filter type` is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, - according to the documentation, the :term:`filter type` is defined as follows: +Step 3: Identify the Triggering Logic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``{Reverse DNS}.{Architecture Subdomain}.student.registration.requested.{Major Version}`` +The triggering logic for the filter should be identified to ensure that the filter is triggered in the right places and that the filter is triggered consistently. We should identify the triggering logic to ensure that maximum coverage is achieved with minimal modifications. The goal is to focus on core, critical areas where the logic we want to modify executes, ensuring the filter is triggered consistently. - Where ``student`` is the subject and ``registration.requested`` the action being performed. The major version is the version of the filter, which will be incremented - when a change is made to the filter that is not backwards compatible, as explained in the ADR. +In our example, the triggering logic could be a place where all enrollment logic goes through. This could be the ``enroll`` method in the enrollment model in the LMS, which is called when a user enrolls in a course in all cases. - Now that you have the :term:`filter type`, you'll need to define the :term:`filter signature` and overall behavior. The :term:`filter signature`, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, - if you're creating a filter that will be used during the student registration process in the LMS, the :term:`filter signature` will be the set of parameters available for that time for the user. In this case, the :term:`filter signature` will be the set of parameters that the registration form sends to the LMS. +.. note:: When designing an filter take into account the support over time of the service and triggering logic. If the service is likely to change or be deprecated, consider the implications of implementing the filter in that service. - You can ask yourself the following questions to help you figure out your filter's parameters: +.. note:: It is helpful to inspect the triggering logic to review the data that is available at the time the filter is triggered. This will help you determine the content of the filter and the data that should be included in the filter arguments. - - What is the filter's purpose? (e.g. to validate the student's email address) - - What parameters will the filter need to to that? (e.g. the email address) - - Where in the registration process will the filter be used? (e.g. after the student submits the registration form but before anything else) +Step 4: Determine the Arguments of the Filter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - With that information, you can define the :term:`filter signature`: +Filter arguments are in-memory objects that can be manipulated and returned to the calling process to change a component's behavior. This is why they depend heavily on the specific behavior you want to modify and the information available at that point in the application flow. It's helpful to ask yourself: - - Arguments: ``email``. Since we want this filter to be broadly used, we'll add as much relevant information as possible for the user at that point. As we mentioned above, we can send more information stored in the registration form like ``name`` or ``username``. - - Returns: since filters take in a set of parameters and return a set of parameters, we'll return the same set of parameters that we received. +- How can this be modified? +- What can I add or change to adjust the behavior? +- Think about the use cases you aim to address. - Since filters also can act according to the result of the filter's execution, we'll need to define the filter's behavior for when the filter stops the process in which it is being used. For example, if you're using the filter in the LMS, you'll need to define - what happens when the filter stops the registration process. So, for this filter we'll define the following behavior: +Our goal is to provide developers with enough control to implement new features while reducing dependencies on the service where the filter is being implemented. However, in some cases, dependencies might be unavoidable, depending on the use case. - - When stopping the registration process, we'll raise a ``PreventRegistration`` exception. +As a rule of thumb, start by passing the most relevant context data from the application flow, and then gradually add more details as you analyze the behavior of the triggering logic. -#. Implement the new filter +.. note:: Consider the criticality of the arguments, could they be removed in the near future? This would mean introducing breaking changes to the filter. -.. Following the steps, you should add the result and any follow-up tasks needed. +In our example, the filter arguments could include the user, course key, and enrollment mode. These arguments are essential for the filter to determine if the user meets the eligibility criteria for enrollment and it is the minimum information required to make the decision (user to check the eligibility, course key to identify the course, and mode to determine the type of enrollment). - Up to this point, you should have the following: +Step 5: Implement the Filter Definition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: python +Implement the :term:`Filter Definition` by creating a new class that inherits from the `OpenEdxPublicFilter`_ class. The filter definition should implement the ``run_filter`` method, which defines the input and output behavior of the filter. The ``run_filter`` method should call the method `run_pipeline`_, passing down the input arguments and returning the final output of the filter. This class should be placed in the appropriate subdomain module in the library, in the ``filters.py`` file. - class StudentRegistrationRequested(OpenEdxPublicFilter): - """ - Custom class used to create registration filters and its custom methods. - """ +.. note:: The input arguments of the ``run_filter`` method should match the arguments that the triggering logic provides. The output of the filter should be consistent with the behavior that the filter intends to modify. Usually, the output is the modified data or the original data if no modifications are needed. - filter_type = "org.openedx.learning.student.registration.requested.v1" +.. note:: Try using type hints to annotate the arguments and return types of the ``run_filter`` method to provide clarity and safety. - class PreventRegistration(OpenEdxFilterException): - """ - Custom class used to stop the registration process. - """ +You can add custom exceptions to the filter to handle specific cases where the filter should halt the application behavior. This will help developers understand when the filter is supposed to halt the application behavior and why. Try not to raise exceptions that are not related to the filter behavior, as this could lead to confusion and unexpected behavior. Only add exceptions if you can justify why the filter should halt the application behavior in that case. - @classmethod - def run_filter(cls, form_data): - """ - Execute a filter with the signature specified. +In our example, the filter definition could be implemented as follows: - Arguments: - form_data (QueryDict): contains the request.data submitted by the registration - form. - """ - sensitive_data = cls.extract_sensitive_data(form_data) - data = super().run_pipeline(form_data=form_data) - return data.get("form_data") +.. code-block:: python -.. note:: - This is not exactly what the registration filter looks like, but it's a good starting point. You can find the full implementation of the registration filter in the library's repository. + class CourseEnrollmentStarted(OpenEdxPublicFilter): + """ + Custom class used to create enrollment filters and its custom methods. + """ - Some things to note: + filter_type = "org.openedx.learning.course.enrollment.started.v1" - - The filter's type is defined in the ``filter_type`` class attribute. In this case, the :term:`filter type` is ``org.openedx.learning.student.registration.requested.v1``. - - The :term:`filter signature` is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. - - The ``run_filter`` is a class method that returns the same set of parameters that it receives. - - The ``run_filter`` class method calls the ``run_pipeline`` method, which is the method that executes the filter's logic. This method is defined in the ``OpenEdxPublicFilter`` class, which is the base class for all the filters in the library. This method returns a dictionary with the following structure: + class PreventEnrollment(OpenEdxFilterException): + """ + Custom class used to stop the enrollment process. + """ - .. code-block:: python + @classmethod + def run_filter(cls, user, course_key, mode): + """ + Execute a filter with the signature specified. - { - "": , - "": , - ... - "": , - } + Arguments: + user (User): is a Django User object. + course_key (CourseKey): course key associated with the enrollment. + mode (str): is a string specifying what kind of enrollment. + """ + data = super().run_pipeline( + user=user, course_key=course_key, mode=mode, + ) + return data.get("user"), data.get("course_key"), data.get("mode") - Where in this specific example would be: +- The ``filter_type`` attribute should be set to the filter type that was identified in the previous steps. +- The ``PreventEnrollment`` class is a custom exception that is raised when the filter should halt the application behavior. +- The ``run_filter`` method is the main method of the filter that is called when the filter is triggered. The method should call the ``run_pipeline`` method, passing down the input arguments and returning the final output of the filter. - .. code-block:: python +Step 6: Trigger the Filter in the Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - { - "form_data": form_data, - } +After implementing the filter definition, you should trigger the filter in the application where the triggering logic is executed. This will ensure that the filter is triggered when the conditions are met and that the filter is modifying the behavior as intended. - Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the :term:`filter pipeline`. That is how ``run_filter`` should always look like. - - The filter's behavior for stopping the process is defined in the ``PreventRegistration`` exception which inherits from the ``OpenEdxFilterException`` base exception. In this case, the exception is raised when the filter stops the registration process. This is done in the service where the filter is being used, which in this case is the LMS. - - The class name is the filter's type ``{Subject}.{Action}`` part in a camel case format. In this case, the filter's name is ``StudentRegistrationRequested``. +In our example, we identified that the triggering logic is the ``enroll`` method in the enrollment model in the LMS. Therefore, we should trigger the filter in the ``enroll`` method, passing down the user, course key, and mode arguments to the filter. The filter should be placed so that it is triggered before the enrollment process is completed, so can alter the enrollment process if the user does not meet the eligibility criteria. -#. Add tests for the new filter +.. note:: Try placing the filter so it can be triggered before the process is completed, so it can alter the process if needed. In some cases, this would be at the beginning of the process, while in others it would be elsewhere. - Each filter has its own set of tests. The tests for the filter you're creating should be located in the ``tests`` directory in the library's repository. The tests should be located in the ``test_filters.py`` file, which is where all the tests for the filters are located. Each set of tests is related to a specific type of filter, so you should add your tests to the set of tests that are related to the filter you're creating. - For example, if you're creating a filter that will be used during the student registration process in the LMS, you should add your tests to the ``TestAuthFilters`` set of tests. This is how the tests for the registration filter look like: +Step 7: Test the Filter +~~~~~~~~~~~~~~~~~~~~~~~ +After triggering the filter in the application, you should test the filter to ensure that it is triggered when the conditions are met and that the filter is modifying the behavior as intended. You should test the filter with different scenarios to ensure that the filter is working as expected and that the filter is not breaking the application by adding tests in the service where the filter is being implemented. Also, test the filter signature by adding unit tests to the library to ensure that the arguments are being passed correctly and that the output is consistent with the behavior that the filter intends to modify. -.. code-block:: python +In the service tests you should include at least the following scenarios: - def test_student_registration_requested(self): - """ - Test StudentRegistrationRequested filter behavior under normal conditions. +- The filter is triggered when the triggering logic is executed. +- The filter when executed with the correct arguments returns the expected output. +- When there are pipeline steps configured, the filter executes the pipeline steps. +- When no pipeline steps are configured, the filter acts as a no-op. +- The filter does not break the application when raising exceptions. - Expected behavior: - - The filter must have the signature specified. - - The filter should return form data. - """ - expected_form_data = { - "password": "password", - "newpassword": "password", - "username": "username", - } - - form_data = StudentRegistrationRequested.run_filter(expected_form_data) - - self.assertEqual(expected_form_data, form_data) - - @data( - ( - StudentRegistrationRequested.PreventRegistration, {"message": "Can't register in this site."} - ), - ) - @unpack - def test_halt_student_auth_process(self, auth_exception, attributes): - """ - Test for student auth exceptions attributes. +You can test the filter by configuring a dummy :term:`Pipeline Step` only for testing purposes. This will allow you to test the filter in isolation and ensure that the filter is working as expected. You can also test the filter in the application by triggering the filter with different scenarios to ensure that the filter is working as expected. In the `test_filters.py`_ you can review how this is done for the enrollment filter. - Expected behavior: - - The exception must have the attributes specified. - """ - exception = auth_exception(**attributes) +Step 8: Implement Your Pipeline Steps +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Filters can be configured with pipeline steps to modify the behavior of the code where the filter is triggered. This allows you to define a sequence of steps that are executed in a specific order to modify the behavior of the application. + +.. TODO: Add a link to the pipeline steps documentation. - self.assertDictContainsSubset(attributes, exception.__dict__) +Step 9: Continue the Contribution Process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. note:: - In this example, we're testing the :term:`filter signature` and the filter's behavior for stopping the process. The first test is testing the :term:`filter signature`, specifically that the behavior works as expected when passed mock form data. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. +After implementing the filter, you should continue the contribution process by creating a pull request in the repository. The pull requests should contain the changes you made to implement the filter, including the filter definition, data attrs, and the places where the filter is triggered. -.. .. seealso:: +For more details on how the contribution flow works, refer to the :doc:`docs.openedx.org:developers/concepts/hooks_extension_framework` documentation. - :ref:`title to link to` +.. _Tutor: https://docs.tutor.edly.io/ +.. _Add Extensibility Mechanism to IDV to Enable Integration of New IDV Vendor Persona: https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4307386369/Proposal+Add+Extensibility+Mechanisms+to+IDV+to+Enable+Integration+of+New+IDV+Vendor+Persona +.. _OpenEdxPublicFilter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L14 +.. _run_pipeline: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L164 +.. _test_filters.py: https://github.com/openedx/edx-platform/blob/master/common/djangoapps/student/tests/test_filters.py#L114-L190 diff --git a/docs/reference/architeture-subdomains.rst b/docs/reference/architecture-subdomains.rst similarity index 100% rename from docs/reference/architeture-subdomains.rst rename to docs/reference/architecture-subdomains.rst diff --git a/docs/reference/filters-configuration.rst b/docs/reference/filters-configuration.rst index e69de29b..e214acdc 100644 --- a/docs/reference/filters-configuration.rst +++ b/docs/reference/filters-configuration.rst @@ -0,0 +1,49 @@ +Filter Configurations +##################### + +The :term:`filter configuration` is a dictionary used to configure the pipeline steps for a particular filter. The configuration settings are specific for each :term:`filter type`. The dictionary looks like this: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "FILTER_TYPE": { # Replace with the specific filter type + "fail_silently": True, # Set to True to ignore exceptions and continue the pipeline + "pipeline": [ + "module.path.PipelineStep0", # Replace with the actual module path and class name + "module.path.PipelineStep1", + # Add more steps as needed + "module.path.PipelineStepN", + ] + }, + } + +Where: + +- ``FILTER_TYPE`` is the :term:`filter type`. +- ``fail_silently`` is a boolean flag indicating whether the pipeline should continue executing the next steps when a runtime exception is raised by a pipeline step. + - If ``True``, when a pipeline step raises a runtime exception (e.g., ``ImportError`` or ``AttributeError``) which are not intentionally raised by the developer during the filter's execution; the exception won't be propagated and the execution will resume, i.e the next steps will be executed. + - If ``False``, the exception will be propagated and the execution will stop returning control to the caller. +- ``pipeline`` is list of paths for each pipeline step. Each path is a string with the following format: ``module.path.PipelineStepClassName``. The module path is the path to the module where the pipeline step class was implemented and the class name is the name of the class that implements the ``run_filter`` method to be executed when the filter is triggered. + +With this configuration: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "FILTER_TYPE": { + "fail_silently": True, + "pipeline": [ + "non_existing_module.PipelineStep", + "existing_module.NonExistingPipelineStep", + "module.path.PipelineStep", + ] + }, + } + +Triggering the filter will behave as follows: + +- The pipeline tooling will catch the ``ImportError`` exception raised by the first step and continue executing the next steps. +- The pipeline tooling will catch the ``AttributeError`` exception raised by the second step and continue executing the next steps. +- The pipeline tooling will execute the third step successfully and then return the result. + +For more details on the configuration see :doc:`../decisions/0002-hooks-filter-config-location`. diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 8d9606a7..ed54f568 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -27,7 +27,7 @@ A filter has multiple components that are used to define, execute and handle fil Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, allowing the filter to control the application flow. E.g., the `CourseEnrollmentStarted filter`_ might raise an exception if the user is ineligible for enrollment called ``PreventEnrollment``. Filter Configuration - Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. E.g., the configuration for the `CourseEnrollmentStarted filter`_ might include settings like ``fail_silently: False`` and ``['my_plugin.filters.StopEnrollmentIfNotValidEmail']`` as its pipeline steps. See the :doc:`/decisions/0002-hooks-filter-config-location` for more details on the configuration format. + The filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. E.g., the configuration for the `CourseEnrollmentStarted filter`_ might include settings like ``fail_silently: False`` and ``['my_plugin.filters.StopEnrollmentIfNotValidEmail']`` as its pipeline steps. See the :doc:`/decisions/0002-hooks-filter-config-location` for more details on the configuration format. This glossary provides a high-level overview of the key concepts and components of the Open edX Filters library. Understanding these terms will help you implement filters in your application and leverage the filter tooling to control the flow of your application based on specific conditions. For a better illustration of these concepts, refer to the :doc:`/how-tos/using-filters` guide. diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 25ab088c..dde0d858 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -7,5 +7,7 @@ References glossary filters + filters-configuration django-plugins-and-filters real-life-use-cases + architecture-subdomains From 197acef07b000adc79138efffbca5dc946bddb12 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 25 Dec 2024 15:38:45 +0100 Subject: [PATCH 04/17] fix: remove trailing whitespace --- docs/how-tos/create-a-new-filter.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index 9097fce8..877a2cb4 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -78,7 +78,7 @@ In our example, the triggering logic could be a place where all enrollment logic Step 4: Determine the Arguments of the Filter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Filter arguments are in-memory objects that can be manipulated and returned to the calling process to change a component's behavior. This is why they depend heavily on the specific behavior you want to modify and the information available at that point in the application flow. It's helpful to ask yourself: +Filter arguments are in-memory objects that can be manipulated and returned to the calling process to change a component's behavior. This is why they depend heavily on the specific behavior you want to modify and the information available at that point in the application flow. It's helpful to ask yourself: - How can this be modified? - What can I add or change to adjust the behavior? From 7f5228fba414016bf2e12f04cc8d3e5fe5888e38 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 25 Dec 2024 15:51:48 +0100 Subject: [PATCH 05/17] docs: mention filter configuration and replace filter content with args --- docs/how-tos/create-a-new-filter.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index 877a2cb4..c3ebfbc2 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -73,7 +73,7 @@ In our example, the triggering logic could be a place where all enrollment logic .. note:: When designing an filter take into account the support over time of the service and triggering logic. If the service is likely to change or be deprecated, consider the implications of implementing the filter in that service. -.. note:: It is helpful to inspect the triggering logic to review the data that is available at the time the filter is triggered. This will help you determine the content of the filter and the data that should be included in the filter arguments. +.. note:: It is helpful to inspect the triggering logic to review the data that is available at the time the filter is triggered. This will help you determine the arguments of the filter and how the filter can modify the behavior. Step 4: Determine the Arguments of the Filter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -95,7 +95,7 @@ In our example, the filter arguments could include the user, course key, and enr Step 5: Implement the Filter Definition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Implement the :term:`Filter Definition` by creating a new class that inherits from the `OpenEdxPublicFilter`_ class. The filter definition should implement the ``run_filter`` method, which defines the input and output behavior of the filter. The ``run_filter`` method should call the method `run_pipeline`_, passing down the input arguments and returning the final output of the filter. This class should be placed in the appropriate subdomain module in the library, in the ``filters.py`` file. +Implement the :term:`filter definition` by creating a new class that inherits from the `OpenEdxPublicFilter`_ class. The filter definition should implement the ``run_filter`` method, which defines the input and output behavior of the filter. The ``run_filter`` method should call the method `run_pipeline`_, passing down the input arguments and returning the final output of the filter. This class should be placed in the appropriate subdomain module in the library, in the ``filters.py`` file. .. note:: The input arguments of the ``run_filter`` method should match the arguments that the triggering logic provides. The output of the filter should be consistent with the behavior that the filter intends to modify. Usually, the output is the modified data or the original data if no modifications are needed. @@ -134,7 +134,7 @@ In our example, the filter definition could be implemented as follows: ) return data.get("user"), data.get("course_key"), data.get("mode") -- The ``filter_type`` attribute should be set to the filter type that was identified in the previous steps. +- The ``filter_type`` attribute should be set to the filter type that was identified in the previous steps. This attribute is used to identify the filter in the :term:`filter configuration`. - The ``PreventEnrollment`` class is a custom exception that is raised when the filter should halt the application behavior. - The ``run_filter`` method is the main method of the filter that is called when the filter is triggered. The method should call the ``run_pipeline`` method, passing down the input arguments and returning the final output of the filter. From 45d94937a4eba33c0a095ce1615bac47dc9685f7 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 25 Dec 2024 15:54:39 +0100 Subject: [PATCH 06/17] refactor: move implement pipeline steps to step 7 --- docs/how-tos/create-a-new-filter.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index c3ebfbc2..17bb7eaf 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -147,7 +147,14 @@ In our example, we identified that the triggering logic is the ``enroll`` method .. note:: Try placing the filter so it can be triggered before the process is completed, so it can alter the process if needed. In some cases, this would be at the beginning of the process, while in others it would be elsewhere. -Step 7: Test the Filter +Step 7: Implement Your Pipeline Steps +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Filters can be configured with pipeline steps to modify the behavior of the code where the filter is triggered. This allows you to define a sequence of steps that are executed in a specific order to modify the behavior of the application. + +.. TODO: Add a link to the pipeline steps documentation. + +Step 8: Test the Filter ~~~~~~~~~~~~~~~~~~~~~~~ After triggering the filter in the application, you should test the filter to ensure that it is triggered when the conditions are met and that the filter is modifying the behavior as intended. You should test the filter with different scenarios to ensure that the filter is working as expected and that the filter is not breaking the application by adding tests in the service where the filter is being implemented. Also, test the filter signature by adding unit tests to the library to ensure that the arguments are being passed correctly and that the output is consistent with the behavior that the filter intends to modify. @@ -162,13 +169,6 @@ In the service tests you should include at least the following scenarios: You can test the filter by configuring a dummy :term:`Pipeline Step` only for testing purposes. This will allow you to test the filter in isolation and ensure that the filter is working as expected. You can also test the filter in the application by triggering the filter with different scenarios to ensure that the filter is working as expected. In the `test_filters.py`_ you can review how this is done for the enrollment filter. -Step 8: Implement Your Pipeline Steps -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Filters can be configured with pipeline steps to modify the behavior of the code where the filter is triggered. This allows you to define a sequence of steps that are executed in a specific order to modify the behavior of the application. - -.. TODO: Add a link to the pipeline steps documentation. - Step 9: Continue the Contribution Process ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From d4c8970266d5dd0f77f9e4b254daaceac02e9f9d Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 25 Dec 2024 16:36:19 +0100 Subject: [PATCH 07/17] docs: clarify identify triggering logic --- docs/how-tos/create-a-new-filter.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index 17bb7eaf..fd6c588c 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -67,9 +67,9 @@ For the enrollment filter, the filter type could be ``org.openedx.learning.cours Step 3: Identify the Triggering Logic ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The triggering logic for the filter should be identified to ensure that the filter is triggered in the right places and that the filter is triggered consistently. We should identify the triggering logic to ensure that maximum coverage is achieved with minimal modifications. The goal is to focus on core, critical areas where the logic we want to modify executes, ensuring the filter is triggered consistently. +The triggering logic for the filter should be identified to ensure that the filter is triggered consistently in the right places. We should ensure that maximum coverage is achieved with minimal modifications when placing the :term:`filter definition` in the service we are modifying. The goal is to focus on core, critical areas where the logic we want to modify executes. -In our example, the triggering logic could be a place where all enrollment logic goes through. This could be the ``enroll`` method in the enrollment model in the LMS, which is called when a user enrolls in a course in all cases. +For this, choose a specific point in the service where the filter should be triggered. This could be a method in a service, a view, or a model where the logic that you want to modify is executed. The triggering logic should be consistent and narrow to ensure that the filter is triggered only when the conditions are met. For instance, the triggering logic should be a place where all enrollment logic goes through, ensuring that the filter is triggered consistently when a user enrolls in a course. This could be the ``enroll`` method in the enrollment model in the LMS, which is called when a user enrolls in a course in all cases. .. note:: When designing an filter take into account the support over time of the service and triggering logic. If the service is likely to change or be deprecated, consider the implications of implementing the filter in that service. From ed218f17e79f325aadde275b2ae4b77ade8c4d42 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 26 Dec 2024 12:01:47 +0100 Subject: [PATCH 08/17] docs: update concepts so it references how-tos index --- docs/concepts/openedx-filters.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/concepts/openedx-filters.rst b/docs/concepts/openedx-filters.rst index 57d37c32..37c7d833 100644 --- a/docs/concepts/openedx-filters.rst +++ b/docs/concepts/openedx-filters.rst @@ -83,9 +83,8 @@ How are Open edX Filters used? Developers can implement functions in an `Open edX Django plugin`_, configure them for a particular filter in the ``OPEN_EDX_FILTERS_CONFIG`` setting, and modify the application flow when a the filter in question is invoked by the process in execution. These functions can the application's behavior by altering data, adding new data, or stopping execution by raising exceptions. For example, a filter can stop a student's enrollment if certain conditions, such as business rules, are not met. -For more information on how to use Open edX Filters, refer to the `Using Open edX Filters`_ how-to guide. +For more information on how to use Open edX Filters, refer to the :doc:`how-tos section <../how-tos/index>`. -.. _Using Open edX Filters: ../how-tos/using-filters.html .. _Hooks Extension Framework: https://open-edx-proposals.readthedocs.io/en/latest/oep-0050-hooks-extension-framework.html .. _Django Signals Documentation: https://docs.djangoproject.com/en/4.2/topics/signals/ .. _CourseEnrollmentStarted filter: https://github.com/openedx/edx-platform/blob/master/common/djangoapps/student/models/course_enrollment.py#L719-L724 From e0f851ac2b4d8b9d9d79e1f9deb21e21a18b4c40 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 19 Dec 2024 19:18:37 +0100 Subject: [PATCH 09/17] docs: add create pipeline step documents --- docs/how-tos/create-a-new-filter.rst | 5 +- docs/how-tos/create-a-pipeline-step.rst | 124 ++++++++++++++++++++++++ docs/how-tos/index.rst | 1 + 3 files changed, 127 insertions(+), 3 deletions(-) create mode 100644 docs/how-tos/create-a-pipeline-step.rst diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index fd6c588c..3a017544 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -137,6 +137,7 @@ In our example, the filter definition could be implemented as follows: - The ``filter_type`` attribute should be set to the filter type that was identified in the previous steps. This attribute is used to identify the filter in the :term:`filter configuration`. - The ``PreventEnrollment`` class is a custom exception that is raised when the filter should halt the application behavior. - The ``run_filter`` method is the main method of the filter that is called when the filter is triggered. The method should call the ``run_pipeline`` method, passing down the input arguments and returning the final output of the filter. +- Use arguments names that are consistent with the triggering logic to avoid confusion and improve readability. Step 6: Trigger the Filter in the Application ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -150,9 +151,7 @@ In our example, we identified that the triggering logic is the ``enroll`` method Step 7: Implement Your Pipeline Steps ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Filters can be configured with pipeline steps to modify the behavior of the code where the filter is triggered. This allows you to define a sequence of steps that are executed in a specific order to modify the behavior of the application. - -.. TODO: Add a link to the pipeline steps documentation. +Implementing pipeline steps allows you to modify the behavior of the application when the filter is triggered. Pipeline steps are a sequence of steps that are executed in a specific order to modify the behavior of the application. You can configure them with the :term:`filter configuration` to define the sequence of steps that are executed when the filter is triggered. Follow the steps in the :doc:`../how-tos/create-a-pipeline-step` guide to implement the pipeline steps for the filter. Step 8: Test the Filter ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst new file mode 100644 index 00000000..dda16bb8 --- /dev/null +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -0,0 +1,124 @@ +Create a Pipeline Step +###################### + +When a filter is triggered in the Open edX platform, the pipeline tooling executes a series of pipeline steps in a specific order. Each pipeline step processes data and returns the output to the next step in the pipeline which can be used to modify the application's behavior. This guide explains how to create a pipeline step for a filter in the Open edX platform. + +Throughout this guide, we will implement the use case of allowing users to enroll in a course only if they have a valid email address. We will create a pipeline step that checks if the user's email address is valid and raise an exception if it is not. + +Assumptions +----------- + +- You have a development environment set up using `Tutor`_. +- You have a basic understanding of Python and Django. +- You understand the concept of filters or have reviewed the relevant :doc:`/concepts/index` docs. +- You are familiar with the terminology used in the project, such as the terms :term:`Filter Type`. If not, you can review the :doc:`../reference/glossary` docs. +- You have reviewed the :doc:`../decisions/0007-filter-design-practices` ADR. +- You have identified that you need to create a new filter and have a use case for the filter. + +Steps +----- + +To create a pipeline step for a filter in the Open edX platform, follow these steps: + +Step 1: Understand your Use Case and Identify the Filter to Use +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before creating a pipeline step, you should understand your use case for the filter and the specific logic you want to implement in the pipeline step. In our example, we want to prevent users from enrolling in a course if they do not have a valid email address. We will create a pipeline step that checks if the user's email address is valid and raise an exception if it is not. + +You should review the list of filters available in the Open edX platform and identify the filter that best fits your use case. In our example, we will use the `CourseEnrollmentStarted filter`_ to implement the logic for our use case. You should review the filter's arguments to understand the data that will be passed to the pipeline step and the expected output. This will help you define the pipeline step's logic and signature. + +Step 2: Install Open edX Filters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, add the ``openedx-filters`` plugin into your dependencies so the library's environment recognizes the event you want to consume. You can install ``openedx-filters`` by running: + +.. code-block:: bash + + pip install openedx-filters + +This will mainly make the filters available for your CI/CD pipeline and local development environment. If you are using the Open edX platform, the library should be already be installed in the environment so no need to install it. + +Step 4: Create a Pipeline Step +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A :term:`pipeline step` is a class that inherits from the base class `PipelineStep`_ and defines specific logic within its `run_filter`_ method. The ``run_filter`` method is executed by the pipeline tooling when the filter is triggered. To create a pipeline step, you should: + +1. Create a new Python module for the pipeline step called ``pipeline.py``. Pipeline steps are usually implemented in a `Django Open edX plugin`_, so you should create the module in the plugin's directory. +2. Create a new class for the pipeline step that inherits from the base class `PipelineStep`_. +3. Implement the logic for the pipeline step within the `run_filter`_ method. The method signature should match the filter's signature to ensure compatibility with the pipeline tooling. In our example, the method should accept the user, course key, and enrollment mode as arguments and return the same arguments if the email address is valid. If the email address is not valid, the method should raise an exception. +4. You can take an iterative approach to developing the pipeline step by testing it locally and making changes as needed. + +In our example, the pipeline step could look like this: + +.. code-block:: python + + from openedx_filters.filters import PipelineStep + + class CheckValidEmailPipelineStep(PipelineStep): + def run_filter(self, user, course_key, mode): + if self.not is_user_email_allowed(user.email): + raise CourseEnrollmentStarted.PreventEnrollment("User does not have a valid email address") + return { + "user": user, + "course_key": course_key, + "mode": mode, + } + +- In this example, we create a new class called ``CheckValidEmailPipelineStep`` that inherits from the base class `PipelineStep`_. +- We implement the logic for the pipeline step within the `run_filter`_ method. The method checks if the user's email address is valid using the ``is_user_email_allowed`` method and raises an exception if it is not. If the email address is valid, the method returns the user, course key, and enrollment mode in a dictionary. +- The method signature matches the filter's signature, accepting the user, course key, and enrollment mode as arguments and returning the same arguments if the email address is valid. You can also return an empty dictionary if you don't need to modify the data. + +Consider the following when creating a pipeline step: + +- Limit each step to a single responsibility to make the code easier to maintain and test. +- Keep the pipeline step logic simple and focused on the specific task it needs to perform. +- Consider the performance implications of the pipeline step and avoid adding unnecessary complexity or overhead, considering the pipeline will be executed each time the filter is triggered. +- Implement error handling and logging in the pipeline step to handle exceptions and provide useful information for debugging, considering both development and production environments. + +Step 5: Configure the Pipeline for the Filter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After creating the pipeline step, you need to configure the pipeline for the filter in the `filter configuration`_. The configuration settings are specific for each :term:`filter type` and define the pipeline steps to be executed when the filter is triggered. You should add the path to the pipeline step class in the filter's pipeline configuration. + +In our example, we will configure the pipeline for the `CourseEnrollmentStarted filter`_ to include the pipeline step we created. The configuration should look like this: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "org.openedx.learning.course.enrollment.started.v1": { + "fail_silently": False, + "pipeline": [ + "my_plugin.pipeline.CheckValidEmailPipelineStep", + ] + }, + } + +Step 6: Test the Pipeline Step +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After creating the pipeline step and configuring the pipeline for the filter, you should test the pipeline step to ensure it works as expected. You can trigger the filter in your development environment and verify that the pipeline step is executed correctly. You should test different scenarios, including valid and invalid email addresses, to ensure the pipeline step behaves as expected. + +You should also implement unit tests for the pipeline step to verify its functionality and handle edge cases. Unit tests can help you identify issues early in the development process and ensure the pipeline step works as intended. To implement the unit test you can directly call the ``run_filter`` method of the filter definition and assert that the pipeline step behaves as expected. Or you can directly call the pipeline step class and assert that the method returns the expected output. + +In our example, you could write a unit test for the pipeline step like this: + +.. code-block:: python + + @override_settings( + OPEN_EDX_FILTERS_CONFIG={ + "org.openedx.learning.course.enrollment.started.v1": { + "fail_silently": False, + "pipeline": [ + "my_plugin.pipeline.CheckValidEmailPipelineStep", + ] + } + } + ) + def test_stop_enrollment_invalid_email(self): + user = UserFactory(email="invalid_email") + with self.assertRaises(CourseEnrollmentStarted.PreventEnrollment): + CourseEnrollmentStarted.run_filter( + user=user, course_key=self.course_key, mode="audit", + ) + +.. _Open edX Django plugin: https://docs.openedx.org/en/latest/developers/concepts/platform_overview.html#new-plugin diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 2325ccf8..fc912645 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -6,3 +6,4 @@ How-tos :caption: Contents: create-a-new-filter + create-a-pipeline-step \ No newline at end of file From 54076bd1c2dd5b32dce104379a9db2b2963ae4d9 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 26 Dec 2024 13:34:20 +0100 Subject: [PATCH 10/17] docs: add new line at the end of index --- docs/how-tos/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index fc912645..f2251e8b 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -6,4 +6,4 @@ How-tos :caption: Contents: create-a-new-filter - create-a-pipeline-step \ No newline at end of file + create-a-pipeline-step From 2d37dbd510b057f2e05dbb274692b9421f2487c6 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 26 Dec 2024 14:03:55 +0100 Subject: [PATCH 11/17] refactor: add missing references to documents --- docs/how-tos/create-a-pipeline-step.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst index dda16bb8..5601b913 100644 --- a/docs/how-tos/create-a-pipeline-step.rst +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -43,7 +43,7 @@ Step 4: Create a Pipeline Step A :term:`pipeline step` is a class that inherits from the base class `PipelineStep`_ and defines specific logic within its `run_filter`_ method. The ``run_filter`` method is executed by the pipeline tooling when the filter is triggered. To create a pipeline step, you should: -1. Create a new Python module for the pipeline step called ``pipeline.py``. Pipeline steps are usually implemented in a `Django Open edX plugin`_, so you should create the module in the plugin's directory. +1. Create a new Python module for the pipeline step called ``pipeline.py``. Pipeline steps are usually implemented in a `Open edX Django plugins`_, so you should create the module in the plugin's directory. 2. Create a new class for the pipeline step that inherits from the base class `PipelineStep`_. 3. Implement the logic for the pipeline step within the `run_filter`_ method. The method signature should match the filter's signature to ensure compatibility with the pipeline tooling. In our example, the method should accept the user, course key, and enrollment mode as arguments and return the same arguments if the email address is valid. If the email address is not valid, the method should raise an exception. 4. You can take an iterative approach to developing the pipeline step by testing it locally and making changes as needed. @@ -78,7 +78,7 @@ Consider the following when creating a pipeline step: Step 5: Configure the Pipeline for the Filter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -After creating the pipeline step, you need to configure the pipeline for the filter in the `filter configuration`_. The configuration settings are specific for each :term:`filter type` and define the pipeline steps to be executed when the filter is triggered. You should add the path to the pipeline step class in the filter's pipeline configuration. +After creating the pipeline step, you need to configure the pipeline for the filter in the :term:`filter configuration`. The configuration settings are specific for each :term:`filter type` and define the pipeline steps to be executed when the filter is triggered. You should add the path to the pipeline step class in the filter's pipeline configuration. In our example, we will configure the pipeline for the `CourseEnrollmentStarted filter`_ to include the pipeline step we created. The configuration should look like this: @@ -121,4 +121,8 @@ In our example, you could write a unit test for the pipeline step like this: user=user, course_key=self.course_key, mode="audit", ) -.. _Open edX Django plugin: https://docs.openedx.org/en/latest/developers/concepts/platform_overview.html#new-plugin +.. _Tutor: https://docs.tutor.edly.io/ +.. _CourseEnrollmentStarted filter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/learning/filters.py#L145-L170 +.. _PipelineStep: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/filters.py#L10-L77 +.. _Open edX Django plugins: https://docs.openedx.org/en/latest/developers/concepts/platform_overview.html#new-plugin +.. _run_filter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/filters.py#L60-L77 From b5c95c7fb37141a2b81029083036a1d169377883 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 7 Jan 2025 17:34:31 +0100 Subject: [PATCH 12/17] docs: add note about implementing multiple exceptions --- docs/how-tos/create-a-new-filter.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index 3a017544..d6b77934 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -139,6 +139,8 @@ In our example, the filter definition could be implemented as follows: - The ``run_filter`` method is the main method of the filter that is called when the filter is triggered. The method should call the ``run_pipeline`` method, passing down the input arguments and returning the final output of the filter. - Use arguments names that are consistent with the triggering logic to avoid confusion and improve readability. +.. note:: Implement exceptions that are related to the filter behavior and specify how the filter should modify the application behavior with each exception. The caller should handle each exception differently based the exceptions purpose. For example, the caller should halt the application behavior when the ``PreventEnrollment`` exception is raised. + Step 6: Trigger the Filter in the Application ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 0be82f185826518d9358c3ccd3669eee821f3562 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 7 Jan 2025 18:12:20 +0100 Subject: [PATCH 13/17] refactor: address PR reviews --- docs/how-tos/create-a-pipeline-step.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst index 5601b913..bfd6ad26 100644 --- a/docs/how-tos/create-a-pipeline-step.rst +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -38,7 +38,7 @@ First, add the ``openedx-filters`` plugin into your dependencies so the library' This will mainly make the filters available for your CI/CD pipeline and local development environment. If you are using the Open edX platform, the library should be already be installed in the environment so no need to install it. -Step 4: Create a Pipeline Step +Step 3: Create a Pipeline Step ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A :term:`pipeline step` is a class that inherits from the base class `PipelineStep`_ and defines specific logic within its `run_filter`_ method. The ``run_filter`` method is executed by the pipeline tooling when the filter is triggered. To create a pipeline step, you should: @@ -54,10 +54,13 @@ In our example, the pipeline step could look like this: from openedx_filters.filters import PipelineStep + # Location my_plugin/pipeline.py class CheckValidEmailPipelineStep(PipelineStep): def run_filter(self, user, course_key, mode): if self.not is_user_email_allowed(user.email): + log.debug("User %s does not have a valid email address, stopping enrollment", user.email) raise CourseEnrollmentStarted.PreventEnrollment("User does not have a valid email address") + log.debug("User has a valid email address, allowing enrollment") return { "user": user, "course_key": course_key, @@ -73,9 +76,9 @@ Consider the following when creating a pipeline step: - Limit each step to a single responsibility to make the code easier to maintain and test. - Keep the pipeline step logic simple and focused on the specific task it needs to perform. - Consider the performance implications of the pipeline step and avoid adding unnecessary complexity or overhead, considering the pipeline will be executed each time the filter is triggered. -- Implement error handling and logging in the pipeline step to handle exceptions and provide useful information for debugging, considering both development and production environments. +- Implement error handling and logging in the pipeline step to handle exceptions and provide useful information for debugging, considering both development and production environments. E.g., when the email is not valid, we raise an exception to prevent the user from enrolling in the course. Logging relevant information when an exception is raised can help identify the root cause of a problem. -Step 5: Configure the Pipeline for the Filter +Step 4: Configure the Pipeline for the Filter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After creating the pipeline step, you need to configure the pipeline for the filter in the :term:`filter configuration`. The configuration settings are specific for each :term:`filter type` and define the pipeline steps to be executed when the filter is triggered. You should add the path to the pipeline step class in the filter's pipeline configuration. @@ -93,7 +96,7 @@ In our example, we will configure the pipeline for the `CourseEnrollmentStarted }, } -Step 6: Test the Pipeline Step +Step 5: Test the Pipeline Step ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After creating the pipeline step and configuring the pipeline for the filter, you should test the pipeline step to ensure it works as expected. You can trigger the filter in your development environment and verify that the pipeline step is executed correctly. You should test different scenarios, including valid and invalid email addresses, to ensure the pipeline step behaves as expected. @@ -104,6 +107,7 @@ In our example, you could write a unit test for the pipeline step like this: .. code-block:: python + # Location my_plugin/tests/test_pipeline.py @override_settings( OPEN_EDX_FILTERS_CONFIG={ "org.openedx.learning.course.enrollment.started.v1": { From 06958287d333f63aecfc46672ec480370ff79365 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 7 Jan 2025 18:15:56 +0100 Subject: [PATCH 14/17] refactor: address PR review about arch subdomains --- docs/reference/architecture-subdomains.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/architecture-subdomains.rst b/docs/reference/architecture-subdomains.rst index c65eb991..dff2befe 100644 --- a/docs/reference/architecture-subdomains.rst +++ b/docs/reference/architecture-subdomains.rst @@ -10,7 +10,7 @@ Currently, these are the `architecture subdomains`_ used by the Open edX Events +-------------------+----------------------------------------------------------------------------------------------------+ | Learning | Allows learners to consume content and perform actions in a learning activity on the platform. | +-------------------+----------------------------------------------------------------------------------------------------+ -| Analytics | Provides insights into learner behavior and course performance. | +| Analytics | Provides visibility into learner behavior and course performance. | +-------------------+----------------------------------------------------------------------------------------------------+ | Enterprise | Provides tools for organizations to manage their learners and courses. | +-------------------+----------------------------------------------------------------------------------------------------+ From 6e81691595d8a7f7cd27f772826701d90f36045e Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 8 Jan 2025 15:08:25 +0100 Subject: [PATCH 15/17] docs: reference understanding the pipeline step logic instead of the filters --- docs/how-tos/create-a-pipeline-step.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst index bfd6ad26..068ce10c 100644 --- a/docs/how-tos/create-a-pipeline-step.rst +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -13,7 +13,7 @@ Assumptions - You understand the concept of filters or have reviewed the relevant :doc:`/concepts/index` docs. - You are familiar with the terminology used in the project, such as the terms :term:`Filter Type`. If not, you can review the :doc:`../reference/glossary` docs. - You have reviewed the :doc:`../decisions/0007-filter-design-practices` ADR. -- You have identified that you need to create a new filter and have a use case for the filter. +- You understand the use case for the filter and the specific logic you want to implement in the pipeline step. Steps ----- From 19d18f63b989228026cd57fcd062e3d7c999599a Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 8 Jan 2025 15:08:50 +0100 Subject: [PATCH 16/17] docs: reference list of filters in docs --- docs/how-tos/create-a-pipeline-step.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst index 068ce10c..ddfa3fcf 100644 --- a/docs/how-tos/create-a-pipeline-step.rst +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -25,7 +25,7 @@ Step 1: Understand your Use Case and Identify the Filter to Use Before creating a pipeline step, you should understand your use case for the filter and the specific logic you want to implement in the pipeline step. In our example, we want to prevent users from enrolling in a course if they do not have a valid email address. We will create a pipeline step that checks if the user's email address is valid and raise an exception if it is not. -You should review the list of filters available in the Open edX platform and identify the filter that best fits your use case. In our example, we will use the `CourseEnrollmentStarted filter`_ to implement the logic for our use case. You should review the filter's arguments to understand the data that will be passed to the pipeline step and the expected output. This will help you define the pipeline step's logic and signature. +You should review the :doc:`list of filters <../reference/filters>` available in the Open edX platform and identify the filter that best fits your use case. In our example, we will use the `CourseEnrollmentStarted filter`_ to implement the logic for our use case. You should review the filter's arguments to understand the data that will be passed to the pipeline step and the expected output. This will help you define the pipeline step's logic and signature. Step 2: Install Open edX Filters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 2ee41e3af51e863fee70b0647c56c15cf899b56d Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Fri, 24 Jan 2025 11:23:55 +0100 Subject: [PATCH 17/17] refactor: address PR reviews --- docs/how-tos/create-a-new-filter.rst | 4 +++- docs/how-tos/create-a-pipeline-step.rst | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/how-tos/create-a-new-filter.rst b/docs/how-tos/create-a-new-filter.rst index d6b77934..c1fcab38 100644 --- a/docs/how-tos/create-a-new-filter.rst +++ b/docs/how-tos/create-a-new-filter.rst @@ -33,6 +33,8 @@ Assumptions - You have reviewed the :doc:`../decisions/0007-filter-design-practices` ADR. - You have identified that you need to create a new filter and have a use case for the filter. +.. warning:: You only need to create a new filter if you have a use case that cannot be implemented using the existing filters. Before creating a new filter, review the :doc:`existing filters <../reference/filters>` to see if any of them can be used to implement your use case. + Steps ----- @@ -49,7 +51,7 @@ Before contributing a new filter, it is important to propose the filter to the c In our example our use case proposal could be: - I want to add a filter that will be triggered when a user enrolls in a course from the course about page. This filter will be used to prfilter users from enrolling in a course if they do not meet the eligibility criteria. The filter will be triggered when the user clicks the enroll button on the course about page and will check if the user meets the eligibility criteria. If the user does not meet the criteria, the filter will raise an exception to prfilter the user from enrolling in the course. + I want to add a filter that will be triggered when a user enrolls in a course from the course about page. This filter will be used to prevent users from enrolling in a course if they do not meet the eligibility criteria. The filter will be triggered when the user clicks the enroll button on the course about page and will check if the user meets the eligibility criteria. If the user does not meet the criteria, the filter will raise an exception to prevent the user from enrolling in the course. If you are confident that the filter is beneficial to the community, you can proceed to the next steps and implement the filter. diff --git a/docs/how-tos/create-a-pipeline-step.rst b/docs/how-tos/create-a-pipeline-step.rst index ddfa3fcf..09a9ff24 100644 --- a/docs/how-tos/create-a-pipeline-step.rst +++ b/docs/how-tos/create-a-pipeline-step.rst @@ -30,7 +30,7 @@ You should review the :doc:`list of filters <../reference/filters>` available in Step 2: Install Open edX Filters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -First, add the ``openedx-filters`` plugin into your dependencies so the library's environment recognizes the event you want to consume. You can install ``openedx-filters`` by running: +First, add the ``openedx-filters`` library into your dependencies so the environment recognizes the filter you want to use. You can install ``openedx-filters`` by running: .. code-block:: bash