Security policies are one of the biggest tasks when working with firewalls. They define how the device handles traffic, whether it lets it through, make it subject to deeper analysis (IDP, AppFW, UTM, etc.) or denies it. It is essential to know the available troubleshooting options and how to use them.
Lets start with short security policies theory recap.
Three categories of security policies exist:
- regular – defined in a zone context (from-zone to-zone). They are unidirectional and evaluated as first.
- global – defined without a zone context and applied to any traffic not handled by regular policies.
- default – the action applied when traffic did not match any regular or global policies. The default “deny” can be changed to “permit” through explicit configuration.
Generally the policy evaluation is done sequentially in the order as they are listed in the configuration. Each security policy consists of a criteria (or condition) and an action. The criteria has following parameters: source address, destination address, application (combination of IP protocol number, source and destination ports, timeout, etc.). Since Junos release 12.1 also source-identity (user role name, any, authenticated-user, unauthenticated-user, unknown-user) is available. The actions are deny, reject and permit. Alternatively the log and count can be defined in the policy action to generate syslogs and maintain policy statistics. Within the permit action further sub-actions can be defined, such as firewall authentication, UTM, IDP, AppFW etc.
Troubleshooting process typically starts with generating the tested traffic if not already present and then examining the session table. Use the „show security flow session“ command to view the session table.
By default the command output will contain ALL sessions on the device. To find the right session information is sometimes looking for needle in a haystack. More importantly in situations where time is of the essence (like taking the JNCIE-SEC exam) it is totally undesired because it takes too long. Fortunately the command output can be narrowed by defining filters. The available filtering options are shown below:
An example:
Combining multiple criteria together is also possible.
The ultimate goal here is to define the criteria that will result in listing only sessions you are looking for.
The output contains information about the session ID, which security policy is handling the session, timeout, interfaces involved and incoming and outgoing bytes/packets statistics. In many cases the shown information suffices. But if needed the „extensive“ parameter reveals further details about the session.
Typically the connection is allowed through the firewall when the session is listed. That does not mean the connection will work because other aspects are involved as well. For instance a session that is listed can hang. Have a look at the packets and bytes counters for both flows. The “0” in the counters for the return flow means the traffic from the responding host does not arrive at the firewall. The problem typically lies somewhere between the firewall and the responding host and prevents it from correct operation (possible reasons could be a routing issue on the way back, another firewall on the path disallowing the connection, or the responding host simply dropping the traffic, etc.).
For situations where the connection is incorrectly handled (such as a session does not appear or wrong security policy is taking care of it or the session appears but it shouldn't, etc.) the next troubleshooting step would be examining the security policies. To do it by looking at the configuration might seem like a good idea. However it can be difficult to spot these kind of errors quickly. A better choice is to use the operational mode command „show security policies“. Similarly as the session command it also allows to define filtering criteria, such as zone context or policy name, etc. , to narrow the output.
Very helpful is the „detail“ parameter because it displays the actual values of the objects (addresses, ports, timeouts) from the policy criteria.
When you are not certain which policies to look at or simply too many would have to be reviewed you can skip examining the policies and go directly to basic data processing traceoptions. A crucial thing to remember is to ALWAYS define a so called “packet filter” for basic data processing traceoptions. It defines criteria for messages that will be placed in the traceoptions file. When omitted ALL traffic processing messages on the device will go in that file. The result will be a huge file. You will spend a lot of time going through it and locating the desired entries. Additionally this vast amount of logging has also a performance impact. Not really something to worry about in the lab environment but in a production environment it is a totally different story. Bottom line – make it a habit always to define the criteria in the “packet-filter” to match the traffic you are troubleshooting.
Below is a configuration example of basic data processing traceoptions:
Once the traceoptions are configured and committed generate the test traffic and then view the log file content. The “show log
- searching for existing session
- destination NAT
- routing lookup
- zones involved
- policy lookup
- source NAT
- etc.
More importantly the reason will be shown why the packet has been permitted or dropped. Do not forget the traceoptions used are for the basic datapath processing only and therefore the file has only information about these operations. Other features (AppFW, IDP, UTM, etc.) have their own traceoptions and logging is done typically in different files. However these features are out of scope for this post.
Below are few example messages from the basic datapath processing traceoptions file. Please note some messages were deleted for brevity.
Example 1: Creating a new session:
Example 2: Packet matching an existing session:
Example 3: Connection denied by a policy:
As mentioned earlier the log file is a very good source to find causes of the problems related to security policies or even routing lookup. And once the cause is determined the only thing remaining is to fix it:-)
For example a different policy selected for handling the traffic than the intended one might be a result of:
- incorrect routing led to different zone context and therefore to evaluating incorrect policies. Correcting the routing resolves the issue.
- incorrect policy sequence. The traffic matched an earlier policy. The solution requires policy reordering .
- incorrect policy criteria. The traffic did either not match the desired policy condition or an earlier policy is incorrectly matching the traffic. Correcting the policy criteria solves the problem.
For the latter two cases use the operational mode command for examining policies described earlier.
Situations exist where you are not able to generate the test traffic. For such cases and for policy reordering the command “show security match-policies” comes very handy. The output lists the policies that match the traffic.
By default only the first policy matching the traffic is shown. However you can override this behavior with the “result-count” parameter and increase the number of matching policies shown (up to 16).
Please keep in mind the command requires all parameters to be defined, otherwise the “missing argument” message appears.
The traceoptions exist also for the security policies configuration related processing, e.g. resolution, compilation, etc.
Below are few example traceoptions messages for security policies configuration processing.
Conclusion
This article presented several possibilities for troubleshooting security policies. At the beginning a short theory recap is provided followed by a description how to use the session table for troubleshooting purposes. A suitable way to examine security policies configuration comes next. This approach tends to be less prone to overlooking mistakes than just looking at the configuration itself. The major part is dedicated to basic datapath traceoptions that reveal details how the firewall processes packets. The “show security match-policies” command together with a brief introduction of policy configuration traceoptions close up this post.
Some tips and hints are included throughout the article that might be helpful to speed up the troubleshooting process during and save time. We all know that saving time is crucial on your JNCIE-SEC exam!
