Html out #531

shmfr · 2023-06-14T09:18:51Z

No description provided.

…ubeSet) holding all connections including src_peers, dst_peers and protocols Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com>

Added optimized_denied_ingress_props and optimized_denied_egress_props (in addition to allowed ones). Improved non_captured_conns computation Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com>

…_HC_set # Conflicts: # nca/CoreDS/CanonicalHyperCubeSet.py

Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com> Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com>

Fixed protocol handling: 0 is a valid protocol number (HOPOPT). Allowing any protocol in the range [0...255], though ProtocolNameResolver does not contain names of all of the possible 256 protocols. Fixed handling non-captured peers in K8S (cannot be handled as denied). Signed-off-by: Tanya <tatyana@il.ibm.com>

…r dst_peers dimensions is all values). Added optimization for fw_rules_map - join different entries having the same values (fw_rules). Signed-off-by: Tanya <tatyana@il.ibm.com>

1. To represent No connections, do not build TcpLikeProperties (TcpLikeProperties with no dimensions represent All connections). 2. Support subsets in query in optimized solution. 3. For comparison of optimized solution to the original one, add connections from peers to themselves. 4. More accurate comparison for 'dot' connectivity queries. 5. Generalized convert_named_ports (to not assume dimensions order) 6. Handling the possibility when projection on one dimension is empty. Signed-off-by: Tanya <tatyana@il.ibm.com>

Properly handling HostEPs in optimized TcpLikeProperties. Signed-off-by: Tanya <tatyana@il.ibm.com>

…perties) Fixed handling of non-captured pods in optimized solution. Added using True/False HC_set (make_all_properties()/make_empty_properties()) Signed-off-by: Tanya <tatyana@il.ibm.com>

'false' - only original run 'true' - only optimized run 'debug'- both runs and comparison of their results. Printing parsing time, queries time and total run time. Signed-off-by: Tanya <tatyana@il.ibm.com>

Workaround for the bug in HC set: using mutual contained_in, instead of == Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com> Signed-off-by: Tanya <tatyana@il.ibm.com>

…for -opt=debug option) Added more debug prints. Better handling of peer_set copying in TcpLikeProperties. Signed-off-by: Tanya <tatyana@il.ibm.com>

Further optimization - calculating ref_ip_blocks only in non-optimized run. Signed-off-by: Tanya <tatyana@il.ibm.com>

Improved comments. Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Tanya <tatyana@il.ibm.com>

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

…ml-out

adisos

added a few initial comments.

add basic tests with output of svg file

adisos · 2023-06-26T08:54:06Z

nca/nca_cli.py

@@ -321,7 +325,7 @@ def nca_main(argv=None):
                        help='A list of labels to subset the query by')
    parser.add_argument('--ghe_token', '--gh_token', type=str, help='A valid token to access a GitHub repository')
    parser.add_argument('--output_format', '-o', type=str,
-                        help='Output format specification (txt, txt_no_fw_rules, csv, md, dot, jpg or yaml). '
+                        help='Output format specification (txt, txt_no_fw_rules, csv, md, dot, jpg, html or yaml). '


add the new html format in the relevant readme files, possibly also include some explanation (and also assumption that graphviz is installed?)

adisos · 2023-06-26T09:01:05Z

nca/FWRules/InteractiveConnectivityGraph.py

@@ -327,6 +393,218 @@ def create_output(self, elements_relations):
                        self._highlight_tag(related_tag, related_tag_info.t_class)
                self._save_tag_file(tag_soup, tag_info)

+        HTML_TEMPLATE = '''


can this template be saved in a separate file?

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

…_HCS

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

adisos · 2023-06-26T10:04:51Z

nca/FWRules/InteractiveConnectivityGraph.py

@@ -59,13 +60,13 @@ class ElementRelations:
        highlights: set = field(default_factory=set)
        explanation: list = field(default_factory=set)

-    def __init__(self, svg_file_name, output_directory):
+    def __init__(self, svg_file_name, output_directory, expl_xml=None):


add the new arg to documentation.

adisos · 2023-07-24T11:38:58Z

nca/Utils/OutputConfiguration.py

@@ -62,6 +64,23 @@ def print_query_output(self, output, supported_output_formats=None):
                    print(f'Command {dot_cmd_string}\n did not create {path}\n', file=sys.stderr)
                if os.path.isfile(tmp_dot_file):
                    os.remove(tmp_dot_file)
+            elif self.outputFormat == 'html':


trying to run the following command from cli:
python -m nca --connectivity -r tests/fw_rules_tests/podlist/ns_list.json -r tests/fw_rules_tests/podlist/pods_list_4.json -r tests/fw_rules_tests/policies/test2-networkpolicy.yaml -o html -f test2.html

but getting:

Failed to create a svg file: test2.html 'NoneType' object is not iterable Command dot test2.html.nca_tmp.dot -Tsvg -otest2.html.nca_tmp.svg did not create test2.html

I do have dot installed on my env

we should avoid the error 'NoneType' object is not iterable even if dot is not installed

Signed-off-by: haim-kermany <haim.kermany@gmail.com>

nca/Parsers/IstioSidecarYamlParser.py

        if alphabet_str:
-            fqdn_regex = r"(?=.{1,254}$)[A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?(\.[A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?)*[.]?"
+            fqdn_regex = "^((?!-)[A-Za-z0-9-]+(?<!-).)+[A-Za-z0-9.]+"


tanyaveksler added 30 commits November 15, 2022 17:42

Initial implementation of the optimized TcpLikeProperties (and HyperC…

74ec605

…ubeSet) holding all connections including src_peers, dst_peers and protocols Signed-off-by: Tanya <tatyana@il.ibm.com>

Extended testcase3 to produce connectivity_map

c279de6

Signed-off-by: Tanya <tatyana@il.ibm.com>

Fixed a problem in HyperCubeSet (wrongly changing self in _and_aux.

1b1ebbe

Added optimized_denied_ingress_props and optimized_denied_egress_props (in addition to allowed ones). Improved non_captured_conns computation Signed-off-by: Tanya <tatyana@il.ibm.com>

Small fix

09cfad1

Signed-off-by: Tanya <tatyana@il.ibm.com>

Further fix of the hyper cube set

94fb117

Signed-off-by: Tanya <tatyana@il.ibm.com>

Further fix of the hyper cube set

520377a

Signed-off-by: Tanya <tatyana@il.ibm.com>

Merge remote-tracking branch 'origin/Optimized_HC_set' into Optimized…

af37b10

…_HC_set # Conflicts: # nca/CoreDS/CanonicalHyperCubeSet.py

Avoiding redundant and heavy copy of layers.

0339215

Signed-off-by: Tanya <tatyana@il.ibm.com>

Merge branch 'master' into Optimized_HC_set

564d667

General changes from the Optimized_HC_set branch.

b9810b3

Signed-off-by: Tanya <tatyana@il.ibm.com>

General changes from the Optimized_HC_set branch.

e82ac99

Signed-off-by: Tanya <tatyana@il.ibm.com>

Avoiding lint warnings

6b3b364

Signed-off-by: Tanya <tatyana@il.ibm.com> Signed-off-by: Tanya <tatyana@il.ibm.com>

Avoiding lint warnings

cfb5ee7

Signed-off-by: Tanya <tatyana@il.ibm.com> Signed-off-by: Tanya <tatyana@il.ibm.com>

Merge branch 'Small-fixes-for-master' into Optimized_HC_set

01a9d40

Added support of IpBlocks in optimized hyper cube set implementation.

28e211b

Signed-off-by: Tanya <tatyana@il.ibm.com>

More accurate update of base_peer_set.

d5d97a6

Signed-off-by: Tanya <tatyana@il.ibm.com>

Added dedundant fw_rules creation for testing (to be further removed).

0a2cb45

Signed-off-by: Tanya <tatyana@il.ibm.com>

Fixed the construction of connectivity graph (when some of src_peer o…

05a496c

…r dst_peers dimensions is all values). Added optimization for fw_rules_map - join different entries having the same values (fw_rules). Signed-off-by: Tanya <tatyana@il.ibm.com>

Properly handling 'False' represented by TcpLikeProperties.

0d4bd28

Properly handling HostEPs in optimized TcpLikeProperties. Signed-off-by: Tanya <tatyana@il.ibm.com>

Added support to ICMP data in optimized HC set (handled in TCPLikePro…

47d5bd0

…perties) Fixed handling of non-captured pods in optimized solution. Added using True/False HC_set (make_all_properties()/make_empty_properties()) Signed-off-by: Tanya <tatyana@il.ibm.com>

Added command line flag optimized_run, having 3 possibilities:

0a8d53f

'false' - only original run 'true' - only optimized run 'debug'- both runs and comparison of their results. Printing parsing time, queries time and total run time. Signed-off-by: Tanya <tatyana@il.ibm.com>

Small fixes;

183eaeb

Workaround for the bug in HC set: using mutual contained_in, instead of == Signed-off-by: Tanya <tatyana@il.ibm.com>

Small fixes;

8e34573

Signed-off-by: Tanya <tatyana@il.ibm.com>

Fixed building tcp_like_properties form connectivity graph

9957228

Signed-off-by: Tanya <tatyana@il.ibm.com> Signed-off-by: Tanya <tatyana@il.ibm.com>

Optimized the comparison between original and optimized connections (…

004742b

…for -opt=debug option) Added more debug prints. Better handling of peer_set copying in TcpLikeProperties. Signed-off-by: Tanya <tatyana@il.ibm.com>

Implemented optimized Istio policy handling.

a27899d

Further optimization - calculating ref_ip_blocks only in non-optimized run. Signed-off-by: Tanya <tatyana@il.ibm.com>

Added Ingress policy support in the optimized solution.

ebae572

Improved comments. Signed-off-by: Tanya <tatyana@il.ibm.com>

Added Istio Ingress policy support in the optimized solution.

25118de

Signed-off-by: Tanya <tatyana@il.ibm.com>

dot header fix

d8e0812

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

shmfr marked this pull request as ready for review June 18, 2023 12:33

shmfr requested a review from adisos June 18, 2023 12:33

shmfr added 8 commits June 20, 2023 14:04

removing explanation for ipblocks

c402d73

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

adding policies in one place

d835df7

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

adding some documentations and test

b7ceb53

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

after merge with master

895d75d

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

updated expected results

dea4134

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

fixed run_all_tests path problem

c4761fc

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

duplicate line removed

fd40627

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

Merge remote-tracking branch 'origin/explainability_with_HCS' into ht…

cef2195

…ml-out

adisos requested changes Jun 26, 2023

View reviewed changes

shmfr added 11 commits June 26, 2023 18:33

updated testes and some docs.

cdaa329

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

little better explanation

658811e

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

some small fixes

7c6ac51

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

base_name added

a468689

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

better documentation for using IP-blocks

224e43b

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

added a ip-block test

cae6397

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

base_name check

7dcfc67

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

base_name check

c33d88a

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

Merge remote-tracking branch 'origin/master' into explainability_with…

43f494b

…_HCS

base_name and sync

2a6e68f

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

after merge with expl branch

f4f7394

Signed-off-by: Shmulik Froimovich <Shmulik.Froimovich@ibm.com>

Base automatically changed from explainability_with_HCS to master July 6, 2023 06:40

adisos reviewed Jul 24, 2023

View reviewed changes

haim-kermany added 2 commits December 5, 2023 12:13

empty merge

0f6cec8

from master

e0b70dd

Signed-off-by: haim-kermany <haim.kermany@gmail.com>

github-advanced-security bot found potential problems Dec 25, 2023

View reviewed changes

adisos closed this Jan 15, 2024

shmfr linked an issue Jan 15, 2024 that may be closed by this pull request

connect interactive map to NCA #456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Html out #531

Html out #531

shmfr commented Jun 14, 2023

adisos left a comment

adisos Jun 26, 2023

adisos Jun 26, 2023

adisos Jun 26, 2023

adisos Jul 24, 2023 •

edited

Loading

Html out #531

Html out #531

Conversation

shmfr commented Jun 14, 2023

adisos left a comment

Choose a reason for hiding this comment

adisos Jun 26, 2023

Choose a reason for hiding this comment

adisos Jun 26, 2023

Choose a reason for hiding this comment

adisos Jun 26, 2023

Choose a reason for hiding this comment

adisos Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

adisos Jul 24, 2023 •

edited

Loading