packages/salesforce: add dashboards

packages/salesforce/_dev/build/docs/README.md

@@ -4,36 +4,35 @@
 
 The Salesforce integration enables users to monitor their [Salesforce](https://www.salesforce.com/) instance effectively. Salesforce is a comprehensive customer relationship management (CRM) platform that supports businesses in managing marketing, sales, commerce, service, and IT teams from a unified platform accessible from anywhere.
 
-### Key Benefits of Salesforce Integration:
+### Key benefits of Salesforce Integration:
 
 - **Operational Insights**: Gain valuable insights into login and logout activities and other operational events within your organization.
 - **Data Visualization**: Create detailed visualizations to monitor, measure, and analyze usage trends and key data, helping you derive actionable business insights.
 - **Proactive Alerts**: Set up alerts to minimize Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR) by referencing relevant logs during troubleshooting.
 
 ## Data streams
 
-The Salesforce integration collects log events using the Salesforce REST API.
+The Salesforce integration comes with the following data streams:
 
-Logs help users maintain a record of events occurring in Salesforce. The log data streams collected by the Salesforce integration include:
+- `login`: Tracks login activity of users who log in to Salesforce.
+- `logout`: Tracks logout activity of users who log out from Salesforce.
+- `apex`: Represents information about various Apex events such as Callout, Execution, REST API, SOAP API, Trigger, etc.
+- `setupaudittrail`: Represents changes users made in the organization's setup area for at least the last 180 days.
+
+The Salesforce integration collects events using the Salesforce REST API. We are collecting following events:
 
 - [Login EventLogFile](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_eventlogfile_login.htm)
 - [Login Platform Events](https://developer.salesforce.com/docs/atlas.en-us.236.0.platform_events.meta/platform_events/sforce_api_objects_logineventstream.htm)
 - [Logout EventLogFile](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_eventlogfile_logout.htm)
 - [Logout Platform Events](https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/sforce_api_objects_logouteventstream.htm)
 - [Apex EventLogFile](https://developer.salesforce.com/docs/atlas.en-us.238.0.object_reference.meta/object_reference/sforce_api_objects_apexclass.htm)
-- [SetupAuditTrail Object](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_setupaudittrail.htm).
-
-Data streams:
-- `login`: Tracks login activity of users who log in to Salesforce.
-- `logout`: Tracks logout activity of users who log out from Salesforce.
-- `apex`: Represents information about various Apex events such as Callout, Execution, REST API, SOAP API, Trigger, etc.
-- `setupaudittrail`: Represents changes users made in the organization's setup area for at least the last 180 days.
+- [SetupAuditTrail Object](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_setupaudittrail.htm)
 
 ## Compatibility
 
-This integration has been tested against Salesforce Spring '22 (v54.0) release.
+This integration has been tested against Salesforce Spring '22 (v54.0) release. Minimum supported version is v46.0.
 
-### Finding Your Salesforce Instance Version
+### Finding your Salesforce Instance Version
 
 You can determine your Salesforce instance version using one of the following methods:
 
@@ -47,9 +46,8 @@ You can determine your Salesforce instance version using one of the following me
 
 1. Use your Salesforce Instance URL with the following format:
    `(Salesforce Instance URL)/services/data`
-2. Example: `https://na9.salesforce.com/services/data`
 
-This will return an XML response listing available API versions. For example:
+Example: `https://na9.salesforce.com/services/data`. This will return an XML response listing with available API versions:
 
 ```xml
 <Versions>
@@ -85,7 +83,7 @@ Ensure that the `API Enabled` permission is selected for the user profile in you
 2. Click on the profile link associated with the `User Account` used for data collection.
 3. Search for the `API Enabled` permission on the profile page. If it’s not present, search under `System Permissions` and check if the `API Enabled` privilege is selected. If not, enable it for data collection.
 
-### Collecting Data Using Real-Time Event Monitoring API
+### Collecting Data using Real-Time Event Monitoring API
 
 To enable data collection using the [Real-Time Event Monitoring API](https://help.salesforce.com/s/articleView?id=sf.real_time_event_monitoring_enable.htm&type=5):
 
@@ -94,29 +92,23 @@ To enable data collection using the [Real-Time Event Monitoring API](https://hel
 3. For each event type you want to monitor (e.g., Login Event, Logout Event), click the dropdown arrow and select "Enable Storage".
 4. Ensure you have the necessary permissions: "View Real-Time Event Monitoring Data" and "Use Real-Time Event Monitoring APIs".
 
-> Note: Real-Time Event Monitoring may require additional licensing. Check with your Salesforce account representative if you're unsure about your subscription level.
+> **Note**: Real-Time Event Monitoring may require additional licensing. Check with your Salesforce account representative if you're unsure about your subscription level.
 
 ## Setup
 
 For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
 
-> **Note:** Please enable either the `login` data stream or the `logout` data stream to avoid data duplication.
-
 ## Configuration
 
-To configure the Salesforce integration in Elastic, you will need the following information from your Salesforce instance:
+To configure the Salesforce integration, you will need the following information from your Salesforce instance:
 
 ### Salesforce Instance URL
 
 The Salesforce Instance URL is the URL of your Salesforce Organization. It can be found in the address bar in Salesforce Classic or Salesforce Lightning.
 
-- **Salesforce Classic**: The value before 'salesforce.com' in the URL is your Salesforce Instance.
+- **Salesforce Classic**: For an example URL `https://na9.salesforce.com/home/home.jsp`, the Salesforce Instance URL is `https://na9.salesforce.com`.
 
-  Example URL: `https://na9.salesforce.com/home/home.jsp`
-
-  In this example, the Salesforce Instance URL is: `https://na9.salesforce.com`
-
-- **Salesforce Lightning**: The instance URL is available under your user name in the “View Profile” tab.
+- **Salesforce Lightning**: The instance URL is available under your user name in the "View Profile" tab.
 
 ### Client Key and Client Secret for Authentication
 
@@ -151,7 +143,7 @@ The User ID of the registered user in Salesforce.
 
 The password used for authenticating the above user.
 
-## Additional Information
+## Additional information
 
 Follow the steps below if you need to find the API version:
 
@@ -171,14 +163,32 @@ Once the Salesforce integration is successfully configured, you can validate the
 
 If the dashboard displays the data correctly, your integration is successfully validated.
 
+## Salesforce Integration: v0.15.0 and Beyond
+
+With version 0.15.0, we've significantly enhanced the Salesforce integration, introducing major changes in data collection mechanisms, authentication, and data streams. Due to these changes, we recommend using Salesforce integration v0.15.0 or above and uninstalling previous versions.
+
+### Key enhancements
+
+1. Unified data collection: The integration now uses a single Filebeat input ([Salesforce input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-salesforce.html)) for data collection from EventLogFile and Real-time Event Monitoring APIs.
+2. JWT authentication: Intoduced JWT authentication mechanism.
+3. Expanded configuration options: Added `initial_interval` and other options to fine-tune data collection, including historical data retrieval.
+4. Change in data-collection mechanism: Replaced Streaming API (cometd) with Real-time Event Monitoring APIs.
+5. Performance optimization: Significantly reduced CPU and memory usage during data collection.
+6. Enhanced dashboards: Improved visualizations.
+
+### Breaking changes
+
+1. Data stream consolidation: Reduced from 6 to 4 data streams — `apex`, `login`, `logout`, and `setupaudittrail`.
+2. Field mapping updates: Modified mappings for multiple fields.
+3. Additional changes: Various other modifications to enhance overall integration performance and functionality.
 
 ## Troubleshooting
 
 This section provides solutions to common issues you might encounter while using the Salesforce integration.
 
 ### Request timeout
 
-If you experience delays in the response from the Salesforce server in the `Apex`, `Login Rest`, `Logout Rest`, or `SetupAuditTrail` data streams, you might encounter the following error:
+If you experience delays in the response from the Salesforce server in the `apex`, `login`, `logout`, or `setupaudittrail` data streams, you might encounter the following error:
 
 ```
 Error while processing http request: failed to execute rf.collectResponse: failed to execute http client.Do: failed to execute http client.Do: failed to read http.response.body
@@ -188,23 +198,9 @@ Error while processing http request: failed to execute rf.collectResponse: faile
 
 ### Data ingestion error
 
-If you encounter data ingestion errors, you might see logs similar to the following:
-
-```json
-{
-    "log.level": "error",
-    "@timestamp": "2022-11-24T12:59:36.835+0530",
-    "log.logger": "input.httpjson-cursor",
-    "log.origin": {
-        "[file.name](http://file.name/)": "compat/compat.go",
-        "file.line": 124
-    },
-    "message": "Input 'httpjson-cursor' failed with: input.go:130: input 8A049E17A5CA661D failed (id=8A049E17A5CA661D)\n\toauth2 client: error loading credentials using user and password: oauth2: cannot fetch token: 400 Bad Request\n\tResponse: {\"error\":\"invalid_grant\",\"error_description\":\"authentication failure\"}",
-    "[service.name](http://service.name/)": "filebeat",
-    "id": "8A049E17A5CA661D",
-    "ecs.version": "1.6.0"
-}
-```
+If you encounter data ingestion errors, you might see an error message similar to the following:
+
+> oauth2 client: error loading credentials using user and password: oauth2: cannot fetch token: 400 Bad Request
 
 **Solution:** Ensure that the `API Enabled` permission is provided to the `profile` associated with the `username` used for the integration. Refer to the **Prerequisites** section above for more information.
 
@@ -214,10 +210,6 @@ If the error persists, follow these steps:
 2. Click on the Connected App name created to generate the client ID and client secret (Refer to Client Key and Client Secret for Authentication) under the Master Label.
 3. Click on `Edit Policies` and select `Relax IP restrictions` from the dropdown for IP Relaxation.
 
-### Missing old events in "Login events table" panel
-
-If **Login events table** does not display older documents after upgrading to version `0.8.0` or later, this issue can be resolved by [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#reindex-with-a-data-stream) the `login` data stream.
-
 ## Logs reference
 
 ### Apex

packages/salesforce/changelog.yml

@@ -1,9 +1,12 @@
 # newer versions go on top
-- version: "0.15.0-next"
+- version: "0.15.0"
   changes:
-    - description: Revamp Salesforce integration to use new filebeat Salesforce input.
-      type: enhancement
+    - description: Revamp Salesforce integration.
+      type: breaking-change
       link: https://github.com/elastic/integrations/pull/9629
+    - description: Add dashboards for the salesforce integration.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/10341
 - version: "0.14.1"
   changes:
     - description: Update README with reindexing steps.

packages/salesforce/data_stream/login/_dev/test/pipeline/test-login-object.log

@@ -1 +1,2 @@
 {"EventDate":"2021-10-19T11:47:22Z","AuthServiceId":null,"CountryIso":"IN","Platform":"Mac OSX","EvaluationTime":0,"CipherSuite":"ECDHE-RSA-AES256-GCM-SHA384","PostalCode":"395007","ClientVersion":"N/A","LoginGeoId":"04F5j00000FadrI","LoginUrl":"login.salesforce.com","LoginHistoryId":"0Ya5j00000GLxCdCAL","CreatedById":"0055j000000q9s7AAA","SessionKey":null,"ApiType":"N/A","AuthMethodReference":null,"LoginType":"Remote Access 2.0","PolicyOutcome":null,"Status":"Success","AdditionalInfo":"{}","ApiVersion":"N/A","EventIdentifier":"06af6d92-1167-467d-a826-ee8583f7134d","RelatedEventIdentifier":null,"LoginLatitude":21.1888,"City":"Surat","Subdivision":"Gujarat","SourceIp":"89.160.20.112","Username":"user.name@email.com","UserId":"0056j000000utlQAAR","CreatedDate":"2021-10-19T11:47:30Z","Country":"India","LoginLongitude":72.8293,"TlsProtocol":"TLS 1.2","LoginKey":"o3vhFaSRBb0OzpCl","Application":"elastic integration","UserType":"Standard","PolicyId":null,"HttpMethod":"POST","SessionLevel":"STANDARD","Browser":"Chrome"}
+{ "ApiType": "N/A", "ApiVersion": "N/A", "Application": "testing_salesforce", "Browser": "Unknown", "CipherSuite": "TLS_AES_256_GCM_SHA384", "ClientVersion": "N/A", "CreatedDate": "2024-07-08T07:26:22.128+0000", "EvaluationTime": 0, "EventDate": "2024-07-08T07:26:18.239+0000", "EventIdentifier": "95eeec6d-1e93-46c1-882b-88bd28f7f8de", "HttpMethod": "POST", "Id": "000000000000000AAA", "LoginHistoryId": "0YaJ400000H0kYoKAJ", "LoginKey": "Qv1P3iAGDrGQxSmN", "LoginType": "Remote Access 2.0", "LoginUrl": "devtestin-dev-ed.develop.my.salesforce.com", "Platform": "Unknown", "SessionLevel": "STANDARD", "SourceIp": "89.160.20.112", "Status": "Failed: Missing Consumer Key Parameter", "TlsProtocol": "TLS 1.3", "UserId": "0055j00000AT6I1AAL", "UserType": "Standard", "Username": "salesforceinstance@devtest.in" }

packages/salesforce/data_stream/login/_dev/test/pipeline/test-login-object.log-expected.json

@@ -13,7 +13,7 @@
                 "created": "2021-10-19T11:47:30.000Z",
                 "dataset": "salesforce.login",
                 "id": "06af6d92-1167-467d-a826-ee8583f7134d",
-                "ingested": "2024-06-21T17:00:24.547548675Z",
+                "ingested": "2024-07-09T13:24:55.645009279Z",
                 "kind": "event",
                 "module": "salesforce",
                 "original": "{\"EventDate\":\"2021-10-19T11:47:22Z\",\"AuthServiceId\":null,\"CountryIso\":\"IN\",\"Platform\":\"Mac OSX\",\"EvaluationTime\":0,\"CipherSuite\":\"ECDHE-RSA-AES256-GCM-SHA384\",\"PostalCode\":\"395007\",\"ClientVersion\":\"N/A\",\"LoginGeoId\":\"04F5j00000FadrI\",\"LoginUrl\":\"login.salesforce.com\",\"LoginHistoryId\":\"0Ya5j00000GLxCdCAL\",\"CreatedById\":\"0055j000000q9s7AAA\",\"SessionKey\":null,\"ApiType\":\"N/A\",\"AuthMethodReference\":null,\"LoginType\":\"Remote Access 2.0\",\"PolicyOutcome\":null,\"Status\":\"Success\",\"AdditionalInfo\":\"{}\",\"ApiVersion\":\"N/A\",\"EventIdentifier\":\"06af6d92-1167-467d-a826-ee8583f7134d\",\"RelatedEventIdentifier\":null,\"LoginLatitude\":21.1888,\"City\":\"Surat\",\"Subdivision\":\"Gujarat\",\"SourceIp\":\"89.160.20.112\",\"Username\":\"user.name@email.com\",\"UserId\":\"0056j000000utlQAAR\",\"CreatedDate\":\"2021-10-19T11:47:30Z\",\"Country\":\"India\",\"LoginLongitude\":72.8293,\"TlsProtocol\":\"TLS 1.2\",\"LoginKey\":\"o3vhFaSRBb0OzpCl\",\"Application\":\"elastic integration\",\"UserType\":\"Standard\",\"PolicyId\":null,\"HttpMethod\":\"POST\",\"SessionLevel\":\"STANDARD\",\"Browser\":\"Chrome\"}",
@@ -85,6 +85,91 @@
                     "name": "Mac OSX"
                 }
             }
+        },
+        {
+            "@timestamp": "2024-07-08T07:26:18.239Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "login-attempt",
+                "category": [
+                    "authentication"
+                ],
+                "created": "2024-07-08T07:26:22.128Z",
+                "dataset": "salesforce.login",
+                "id": "95eeec6d-1e93-46c1-882b-88bd28f7f8de",
+                "ingested": "2024-07-09T13:24:55.645018952Z",
+                "kind": "event",
+                "module": "salesforce",
+                "original": "{ \"ApiType\": \"N/A\", \"ApiVersion\": \"N/A\", \"Application\": \"testing_salesforce\", \"Browser\": \"Unknown\", \"CipherSuite\": \"TLS_AES_256_GCM_SHA384\", \"ClientVersion\": \"N/A\", \"CreatedDate\": \"2024-07-08T07:26:22.128+0000\", \"EvaluationTime\": 0, \"EventDate\": \"2024-07-08T07:26:18.239+0000\", \"EventIdentifier\": \"95eeec6d-1e93-46c1-882b-88bd28f7f8de\", \"HttpMethod\": \"POST\", \"Id\": \"000000000000000AAA\", \"LoginHistoryId\": \"0YaJ400000H0kYoKAJ\", \"LoginKey\": \"Qv1P3iAGDrGQxSmN\", \"LoginType\": \"Remote Access 2.0\", \"LoginUrl\": \"devtestin-dev-ed.develop.my.salesforce.com\", \"Platform\": \"Unknown\", \"SessionLevel\": \"STANDARD\", \"SourceIp\": \"89.160.20.112\", \"Status\": \"Failed: Missing Consumer Key Parameter\", \"TlsProtocol\": \"TLS 1.3\", \"UserId\": \"0055j00000AT6I1AAL\", \"UserType\": \"Standard\", \"Username\": \"salesforceinstance@devtest.in\" }",
+                "outcome": "failure",
+                "provider": "Object",
+                "type": [
+                    "info"
+                ],
+                "url": "devtestin-dev-ed.develop.my.salesforce.com"
+            },
+            "http": {
+                "request": {
+                    "method": "POST"
+                }
+            },
+            "related": {
+                "ip": [
+                    "89.160.20.112"
+                ]
+            },
+            "salesforce": {
+                "login": {
+                    "api": {
+                        "type": "N/A",
+                        "version": "N/A"
+                    },
+                    "application": "testing_salesforce",
+                    "client_version": "N/A",
+                    "evaluation_time": 0.0,
+                    "history_id": "0YaJ400000H0kYoKAJ",
+                    "key": "Qv1P3iAGDrGQxSmN",
+                    "type": "Remote Access 2.0"
+                }
+            },
+            "source": {
+                "geo": {
+                    "city_name": "Linköping",
+                    "continent_name": "Europe",
+                    "country_iso_code": "SE",
+                    "country_name": "Sweden",
+                    "location": {
+                        "lat": 58.4167,
+                        "lon": 15.6167
+                    },
+                    "region_iso_code": "SE-E",
+                    "region_name": "Östergötland County"
+                },
+                "ip": "89.160.20.112"
+            },
+            "tags": [
+                "preserve_original_event"
+            ],
+            "tls": {
+                "cipher": "TLS_AES_256_GCM_SHA384",
+                "version": "1.3",
+                "version_protocol": "tls"
+            },
+            "user": {
+                "email": "salesforceinstance@devtest.in",
+                "id": "0055j00000AT6I1AAL",
+                "roles": [
+                    "Standard"
+                ]
+            },
+            "user_agent": {
+                "name": "Unknown",
+                "os": {
+                    "name": "Unknown"
+                }
+            }
         }
     ]
 }

packages/salesforce/data_stream/login/elasticsearch/ingest_pipeline/object.yml

@@ -156,7 +156,7 @@ processors:
       target_field: source.geo
       ignore_missing: true
       ignore_failure: true
-      if: 'ctx?.source?.ip != null && !ctx.source?.geo?.location?.containsKey("lat") && !ctx.source?.geo?.location?.containsKey("lon")'
+      if: 'ctx?.source?.ip != null && ctx.source?.geo?.location?.lat == null && ctx.source?.geo?.location?.lon == null'
   - rename:
       field: json.CountryIso
       target_field: source.geo.country_iso_code