dnsdbflex.man

.\" Copyright (c) 2020 by Farsight Security, Inc.
.\"
.\" Licensed under the Apache License, Version 2.0 (the "License");
.\" you may not use this file except in compliance with the License.
.\" You may obtain a copy of the License at
.\"
.\"  http://www.apache.org/licenses/LICENSE-2.0
.\"
.\" Unless required by applicable law or agreed to in writing, software
.\" distributed under the License is distributed on an "AS IS" BASIS,
.\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.\" See the License for the specific language governing permissions and
.\" limitations under the License.
.Dd 2020-07-31
.Dt dnsdbflex 1 DNSDB
.Os " "
.Sh NAME
.Nm dnsdbflex
.Nd DNSDB flexible query tool
.Sh SYNOPSIS
.Nm dnsdbflex
.Op Fl cdFjhqTUv46
.Op Cm --exclude Ar glob|regular_expression
.Op Cm --force
.Op Cm --glob Ar glob
.Op Cm --mode Ar terse
.Op Cm --regex Ar regular_expression
.Op Cm --timeout Ar timeout
.Op Fl A Ar timestamp
.Op Fl B Ar timestamp
.Op Fl l Ar query_limit
.Op Fl O Ar offset
.Op Fl s Ar search_what
.Op Fl t Ar rrtype
.Op Fl u Ar server_sys
.Sh DESCRIPTION
.Nm dnsdbflex
constructs and issues flexible search queries to Farsight Security's
DNSDB system.  The flexible searches include regular expressions and globs (i.e. full wildcarding).  The results from
.Nm dnsdbflex
can be displayed directly as JSON or emitted in the
.Nm dnsdbq
batch file input format.  Using the batch file format allows "pivoting" from an flexible search into complete DNSDB results.
.Pp
Values to
.Nm --glob,
.Nm --regex,
or
.Nm --exclude
are called search expressions.
.Pp
Search expressions must match the the data indexed in DNSDB's flexible
search database.  All DNS rrnames end in a dot.  All rdata
indexed in the database will end in a dot if it's a host name or will
end in a double quote for other data.  A search expression that does
not conform with that may be a wasted query.  For example,
.Nm dnsdbflex --glob '*.fsi.io'
will not match anything because globs are right-anchored and that
search expression does not end in a dot.  In glob search expressions,
.Nm dnsdbflex
normally detects such violations and disallows them.  Add a
trailing dot to match something,
.Nm dnsdbflex --glob '*.fsi.io.'
.Pp
.Nm dnsdbflex
doesn't detect such violations in regex search expressions.
For example,
.Nm dnsdbflex --regex '.*\\.fsi\\.io$'
will not match anything but
.Nm dnsdbflex --regex '.*\\.fsi\\.io\\.$'
does (note the dot before the dollar sign).
.Pp
Search expressions must contain 7 bit clean, printable ASCII
characters.  Use Punycode IDN encoding to search for IDN domain names.
Use \\DDD (where DDD is the decimal value of the character) to match
non-printable characters in rdata strings.
.Pp
See documentation referenced by <\fI\%https://api.dnsdb.info\fP> for a description of the Farsight Compatible Regular Expression (FCRE) syntax and the full wildcarding syntax, as well as more examples.
.Pp
You'll need to get an API key from Farsight to use
.Nm dnsdbflex
with DNSDB.  Note that certain types of API keys may not be allowed to use this API, in which case,
.Nm dnsdbflex
will fail with error message "The type of API key is not allowed to
use the DNSDB Flex API".
.Sh OPTIONS
.Bl -tag -width 3n
Either
.Nm --glob
or
.Nm --regex
must be specified. Both cannot be specified at the same time.
.It Cm --exclude Ar glob|regular_expression
Filters out results selected by a glob or regular expression.
If
.Nm --glob
was specified, then
.Nm --exclude
takes a glob.
If
.Nm --regex
was specified, then
.Nm --exclude
takes a regular expression.
.It Cm --force
Issue search queries even if rejected by
.Ic dnsdbflex's
pattern checks.
.It Cm --glob Ar glob
Specify that
.Nm dnsdbflex
should do a glob search.
Only the * and [] glob operators are supported.  Can abbreviate as
.Ic --g .
.It Cm --mode Ar terse
Specify mode of information to return in results.
.Bl -tag -width Ds
.It Cm terse
Can abbreviate as 't'.  This is the only value currently supported and
it is the default, so the
.Fl -mode
option need not be specified.
.Pp
For rrnames queries, returns the rrname and rrtype.
.Pp
For rdata queries, returns normalized rdata, rrtype, and raw_rdata.
.El
.It Cm --regex Ar regular_expression
Specify that
.Nm dnsdbflex
should do a regular expression search in the FCRE syntax.  Can abbreviate as
.Ic --r .

.It Cm --timeout Ar timeout
Specify the timeout, in seconds, for the initial connection to the database server and for each subsequent transaction. 0 means no timeout.

.It Fl A Ar timestamp
Specify a backward time fence. Only results seen by the passive DNS
sensor network on or after this time will be selected. See also
.Fl c .
See the TIME FENCING section for more information.
.It Fl B Ar timestamp
Specify a forward time fence. Only results seen by the passive DNS
sensor network on or before this time will be selected. See also
.Fl c .
See the TIME FENCING section for more information.
.It Fl c
By default,
.Fl A
and
.Fl B
(separately or together) will select partial overlaps of database tuples and
time search criteria. To match only complete overlaps, add the -c
("completeness") command line option (this is also known as "strict"
mode).  See the TIME FENCING section for more information.
.It Fl d
enable debug mode.  Repeat for more debug output.
.It Fl F
specify batch output mode, outputting results in the batch format that
.Nm dnsdbq -f
can read.
.Fl F
includes the rrtype in the batch file format queries -- in contrast to
.Fl T
described below.
.Pp
If searching for rdata, if an rdata value is not printable or contains
whitespace, it will emit it using the raw_rdata hex value and output a
comment line giving the non-raw format.
.Pp
See
.Xr dnsdbq 1
for documentation of the batch format.
.It Fl h
emit usage and quit.
.It Fl j
output in JSON format, which is the default.
.Fl j
and
.Fl F
are mutually exclusive.
.It Fl l Ar query_limit
query for that limit's number of responses. If specified as 0 then the DNSDB
API server will return the maximum limit of results allowed.  If
.Fl l ,
is not specified, then the query will not specify a limit, and the DNSDB API
server may use its default limit.
.It Fl O Ar offset
to offset by #offset the results returned by the query.  This gives
you approximate incremental results transfers.  Results can be
reordered between queries, so using progressively higher offsets is
not guaranteed to return all results without duplicates or gaps.
Offset cannot be negative and the default is 0.
.It Fl q
(quiet) suppresses sending warning messages to stderr.
.It Fl s Ar rrnames|rdata
what data to search.
.Bl -tag -width Ds
.It Cm rrnames
Search in the rrnames part of DNS records, aka the left-hand side.  Can abbreviate as 'n'.  This is the default.
.It Cm rdata
Search in the rdata part of DNS records, aka the right-hand side.  Can abbreviate as 'd'.
.El
.It Fl t Ar rrtype
specify the resource record type desired.  Default is ANY.
.Pp
For rrnames queries, valid
rrtypes include those defined in DNS RFCs, including ANY, except DNSSEC
types are not allowed (ANY-DNSSEC, CDNSKEY, CDS, DLV, DNSKEY, DS,
NSEC, NSEC3, NSEC3PARAM, and RRSIG resource record types).  Also valid
are TYPE# values.
.Pp
For rdata queries, only the following rrtypes are valid: CNAME,
HINFO, MX, NAPTR, NS, PTR, RP, SOA, SPF, SRV, and TXT.  Also valid are
their TYPE# values.
.It Fl T
Like
.Fl F
but does not include the rrtype in the batch file queries.
This allows pivots to match against all available rrtypes.  The batch
output will also include a comment for each line including the rrtype.
.It Fl u Ar server_sys
specifies the syntax of the RESTful URL.  The only system currently
supported is "dnsdb2", which is the default.
.It Fl U
turns off TLS certificate verification (unsafe).
.It Fl v
report the version of
.Nm dnsdbflex
 and exit.
.It Fl 4
force connecting to the DNSDB server via IPv4.
.It Fl 6
force connecting to the DNSDB server via IPv6.
.El
.Sh EXAMPLES
.Pp
.Bd -literal -offset 2n
# Regular expression search of all rrnames that contain a coke label,
# for all rrtypes, limit of 10 results.
$ dnsdbflex --regex '.*\\.coke\\..*' -l 10

# Same query without using default values
$ dnsdbflex --regex '.*\\.coke\\..*' -l 10 -s rrnames --mode terse

# Glob search of all names that contain a coke label and have an 'A' RRType.
$ dnsdbflex --glob '*.coke.*' -l 10 -t A

# Pivot those results into dnsdbq for full DNSDB API results in json
# form.  Note that up to 11 DNSDB query quota units will be consumed,
# 1 by dnsdbflex and 10 by dnsdbq.  If we did not specify the RRType
# to dnsdbflex, then it might return more than 10 results (one for
# each RRType for each name) and we'd use more than 11 DNSDB query
# quota units.
$ dnsdbflex --glob '*.coke.*' -l 10 -t A -F | dnsdbq -f -j

# Get names containing "coke" but then exclude all those containing "diet".
$ dnsdbflex --glob '*.coke.*' --exclude '.*diet.*' -l 10

# Same query, but using regular expressions
$ dnsdbflex --regex '.*\\.coke\\..*' --exclude '.*\\.diet\\..*' -l 10
.Ed
.Pp
.Sh "TIME FENCING"
Farsight's DNSDB flexible search provides time fencing options for
searches.  The
.Fl A
and
.Fl B
options take a timestamp as an argument.  The timestamps may be one of
following forms.
.Bl -dash -offset indent
.It
positive unsigned integer : in Unix epoch format.
.It
negative unsigned integer : negative offset in seconds from now.
.It
YYYY-MM-DD [HH:MM:SS] : in absolute form, in UTC time, as DNSDB does its
fencing using UTC time.
.It
%uw%ud%uh%um%us : the relative form with explicit labels (w=weeks, d=days,
h=hours, m=minutes, s=seconds).  Calculates offset
from UTC time, as DNSDB does its fencing using UTC time.
.El
.Pp
A few examples of how to use time fencing options:
.Bd -literal -offset 4n
# Responses after Aug 22, 2015 (midnight),
# excluding records ALSO seen before that time.
$ dnsdbflex... -c -A 2015-08-22

# Responses from 2015 (midnight to midnight),
# but not excluding records ALSO seen outside that time range.
$ dnsdbflex... -B 2016-01-01 -A 2015-01-01
.Ed
.Pp
Certain settings for time fences may be used to accelerate
queries for rrnames and rdata values which have been recently observed
or which were first observed in the distant past.  Time fencing may
accelerate the query if either
.Fl A
or
.Fl B
(but not both) are supplied without
.Fl c .
.Pp
A few examples of how to use time fencing options where the query
may be accelerated:
.Bd -literal -offset 4n
# Responses after 2015-08-22 14:36:10,
# but not excluding records ALSO seen before that time.
$ dnsdbflex... -A "2015-08-22 14:36:10"

# Responses from the last 60 minutes,
# but not excluding records ALSO seen before that time.
$ dnsdbflex... -A "-3600"

# Responses after Aug 22, 2015 (midnight),
# but not excluding records ALSO seen before that time.
$ dnsdbflex... -A 2015-08-22

# Responses before Jan 22, 2013 (midnight),
# but not excluding records ALSO seen after that time.
$ dnsdbflex... -B 2013-01-22
.Ed
.Sh FILES
.Ic ~/.dnsdb-query.conf
or
.Ic /etc/dnsdb-query.conf :
configuration file which can specify the API key, etc. variables. The
first of these which is readable will be used, alone, in its
entirety.  See the
.Ic DNSDBQ_CONFIG_FILE
environment variable which can specify a different configuration
file to use.
.Pp
For backwards compability,
.Ic ~/.isc-dnsdb-query.conf
and
.Ic /etc/isc-dnsdb-query.conf
are also valid, but deprecated.
.Pp
The variables which can be set in the configuration file are as
follows:
.Bl -tag -width ".Ev DNSDB_API_KEY , APIKEY"
.It Ev DNSDB_API_KEY , APIKEY
contains the user's DNSDB apikey (no default).
.It Ev DNSDB_SERVER
contains the URL of the DNSDB API server (default is <\fI\%https://api.dnsdb.info\fP>),
and optionally the URI prefix for the database.
.It Ev DNSDBQ_SYSTEM
contains the default value for the
.Ar u
option described above. Can only be "dnsdb2". If unset,
.Nm dnsdbflex
will probe for any configured system.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
.Nm :
.Bl -tag -width ".Ev DNSDBQ_CONFIG_FILE"
.It Ev DNSDBQ_CONFIG_FILE
specifies the configuration file to use, overriding the internal search list.
.It Ev DNSDB_API_KEY
contains the user's apikey. The older APIKEY environment variable has
been retired, though it can still be used in the configuration file.
.It Ev DNSDB_SERVER
contains the URL of the DNSDB API server, and optionally a URI prefix to be
used. If not set, the configuration file is consulted.
.It Ev DNSDBQ_SYSTEM
See DNSDBQ_SYSTEM in the FILES section above.
.It Ev DNSDBFLEX_TIMEOUT
Specify the timeout (in seconds) to use for network requests. An alternative to passing
.Ic --timeout <seconds>
on the command line
.El
.Sh "EXIT STATUS"
Success (exit status zero) occurs if a connection could be established
to the back end database server, even if no records matched the search
criteria. Failure (exit status nonzero) occurs if no connection could be
established, perhaps due to a network or service failure, or a configuration
error such as specifying the wrong server hostname.
.Sh "SEE ALSO"
.Xr dnsdbq 1 ,
.Xr jq 1 ,
.Xr libcurl 3 ,
.Xr <\fI\%https://api.dnsdb.info/\fP>